I want to explain detail the following paragraph: AI (from: https://www.anthropic.com/research/provable-inference)

 



  1. For a large model (Llama 4 Maverick, 400B-parameter mixture of experts), a single H100 GPU generates proofs at around 110 tokens/second, with a proof size of ~19 MB per 1,024-token request, when in 4-bit quantized weights, using integer arithmetic in prefill. This is ~70-95x the GPU time of running the model itself on vLLM, a common open-source library.


Explained:  

The paragraph describes a significant development in cryptographically verifiable AI inference: researchers have demonstrated that a system called FlashProve can generate mathematical proofs that a very large language model executed a specified computation correctly, without revealing its internal parameters.

However, generating those proofs is substantially more computationally expensive than performing ordinary AI inference.

The most important distinction is that 110 tokens/second refers to the rate at which FlashProve processes tokens for proof generation, not the rate at which the language model generates new text.

I examined Anthropic's November 2, 2026 publication, its technical report, and the project's performance documentation. The details below clarify the experimental results and their implications.

1. Understanding the experimental configuration

FLASHPROVE BENCHMARK

400B

Total model parameters

110

Prompt tokens processed/s

18.78 MB

Proof size per request

70–95×

GPU-time overhead vs. vLLM
One NVIDIA H100 80 GB GPU, 24 host CPU cores, 4-bit model weights, 1,024-token contexts and batched proof generation.

What does a 400B-parameter model mean?

The benchmark uses Llama 4 Maverick developed by Meta.

The model contains approximately 400 billion parameters. Parameters are numerical values learned during training that determine how the network processes information and generates predictions.

A conventional dense model might use essentially all of its parameters during each inference operation.

Llama 4 Maverick instead uses a Mixture of Experts (MoE) architecture.

An MoE model contains multiple specialized neural-network components called experts. A routing mechanism selects which experts participate in processing each token.

In this particular model:

  • Total parameters: approximately 400 billion.

  • Experts: 128.

  • Parameters active per token: approximately 17 billion.

Consequently, the model has a very large total capacity, but a smaller subset of its parameters participates in processing each token.

This is particularly important for FlashProve because its cryptographic proving workload can exploit the sparse activation of experts. The prover does not need to perform the same active computation across all 400 billion parameters for every token.

Nevertheless, all model weights must be committed to cryptographically so that an unauthorized substitution cannot go undetected.


2. What does 4-bit quantization mean?

Neural networks ordinarily represent their parameters using numerical formats such as FP32, FP16 or BF16.

For instance, a neural-network parameter might be represented approximately as:

 w=0.734582

A 32-bit representation uses four bytes for each parameter. A 16-bit representation uses two bytes.

In 4-bit quantization, each parameter is represented using only four bits, typically as part of a quantized representation involving scaling factors.

That allows 16 discrete stored codes per quantization group or format, with scaling used to approximate a wider range of numerical values.

How much memory can quantization save?

Representation

Theoretical storage for 400B parameters

FP32 (32-bit)

1,600 GB

FP16 (16-bit)

800 GB

FP8 (8-bit)

400 GB

INT4 (4-bit)

200 GB

These are idealized figures for raw parameters. Actual memory requirements are larger because quantized models may also require scales, metadata, embeddings and other storage.

Anthropic reports that its quantized Llama 4 Maverick model occupied approximately 232 GB of host memory.

This explains why quantization is essential for making cryptographic inference proofs computationally manageable.

Why does FlashProve use integer arithmetic?

This is an especially important technical detail.

Ordinary neural networks often execute floating-point mathematical operations.

Floating-point arithmetic may introduce slight differences because of rounding, hardware execution order or implementation details.

For a cryptographic proof, these small differences can become problematic.

The prover must establish that a specific computation produced a specific result.

FlashProve therefore uses an integer-based representation of the quantized model. Integer arithmetic is easier to specify exactly and connect to the algebraic constraints used in cryptographic proof systems.

For example, a neural-network operation can be represented in simplified form as:

 y= Sumatoria de i=1 a N (WiXi +b)  

When weights, activations and scaling rules are represented through precisely specified integer operations, the prover can establish that the required computational relationships were satisfied.

Importantly, FlashProve proves the behavior of this defined integer model, not necessarily the exact numerical behavior of an arbitrary floating-point implementation.

This distinction matters because quantization and numerical approximations can change which output token receives the highest score.

3. What does "prefill" mean?

Large language models typically have two main inference stages.

Stage 1

Prefill

The model processes the input prompt, often evaluating many input-token positions in parallel, and constructs the information needed to predict subsequent tokens.
Stage 2

Decode

The model generates new tokens autoregressively, using previous tokens and the stored context to produce the next token.

Suppose a user submits a prompt containing 1,024 tokens and requests a 200-token answer.

During prefill, the model processes the original 1,024-token prompt.

During decoding, the model generates the 200 new tokens, normally one after another.

The distinction matters because prefill and decoding have different computational characteristics.

The FlashProve figure of 110 tokens/second is a prefill-equivalent processing rate. It is not a decoding throughput measurement.

In the principal benchmark, each proof certifies one next-token prediction following a 1,024-token context. Specifically, it certifies the token selected by greedy decoding—the token with the highest model score.

This means the benchmark does not demonstrate that FlashProve can certify 110 newly generated answer tokens every second.

The technical report also discusses a separate experiment verifying complete stored replies, but that is a different measurement.


4. How can one H100 GPU handle a 232 GB model?

An NVIDIA H100 with 80 GB of GPU memory cannot store the entire 232 GB model at once.

FlashProve resolves this by keeping the model weights in the server's ordinary RAM and transferring the necessary portions into GPU memory as required.

This is sometimes called weight streaming or layer-wise execution.

The implementation processes the model in segments rather than attempting to hold every model parameter and intermediate computation simultaneously in GPU memory.

It also processes proof-related intermediate data in segments.

This is a memory-management technique, not evidence that the 400B model has somehow been compressed to 80 GB.

According to the technical report, the benchmark requires a single H100 GPU plus substantial host RAM and CPU resources. Therefore, the phrase "a single H100" should not be interpreted as meaning that the experiment runs on a GPU alone.

5. What exactly does 110 tokens/second mean?

This number is best understood through the actual benchmark.

FlashProve processes a batch of 24 requests. Each request contains a 1,024-token context.

The reported time for processing the batch and producing proofs was approximately 223 seconds.

Therefore, the reported throughput corresponds to approximately 110 processed context-token positions per second, aggregated across the batch.

This implies an amortized processing time of approximately 9.3 seconds per request.

However, that is not the actual waiting time for an individual request.

The batch of 24 requests takes about 223 seconds before its proofs are ready.

The performance documentation also reports that producing one proof without batching takes approximately 59 seconds, with substantially lower overall throughput.

This is a classic throughput-versus-latency trade-off.

Batching improves total efficiency, but individual requests may wait longer for their results.

6. Why is each cryptographic proof approximately 19 MB?

A proof is not a copy of the model, a record of all its parameters, or a complete log of every intermediate activation.

Instead, it contains cryptographic evidence that specified computational relationships were satisfied.

FlashProve uses techniques from the zkSNARK family, including commitments, algebraic checks, cryptographic hashes and proof composition.

Conceptually, the provider first commits to a particular collection of model weights.

For a given prompt, it then produces a token and a proof that the token was computed according to the specified model.

The verifier checks a mathematical relation of the general form:

Verify (C,x,y,pi)=1 

where:

Symbol

Meaning

\(C\)

Commitment to the model weights

\(x\)

Input tokens

\(y\)

Claimed output token

\(\pi\)

Cryptographic proof

1

Verification succeeds

The important property is that the verifier does not need to execute the 400B-parameter model.

It only needs to check the supplied evidence against the commitment and the claimed input and output.


Why does 19 MB matter?

The size is remarkably small relative to the scale of the computation being certified.

The model weights occupy approximately 232 GB, while a proof occupies 18.78 MB.

But there is an important practical limitation: a short textual response might occupy only a few kilobytes, whereas its associated proof could require almost 19 MB of additional data.

For an application handling thousands or millions of requests, the bandwidth and storage requirements become significant.

Anthropic reports that CPU verification takes approximately 17 seconds using 24 threads or 21 seconds on one thread under the stated experimental conditions.

Thus, the cryptographic proof is much cheaper to verify than recomputing the large model, but it is not yet a negligible-cost operation.

7. Why is FlashProve 70–95 times slower than ordinary inference?

This is arguably the most important performance result.

Ordinary inference asks:

What output does this neural network produce?

Provable inference asks a more demanding question:

Can we produce cryptographic evidence convincing an independent verifier that the specified neural network produced this output, without revealing its weights?

The second problem requires substantially more work.

In addition to carrying out the model computation, the prover must construct and check representations of the intermediate operations, create cryptographic commitments, generate algebraic proof material and produce a verifier-checkable proof.

The meaning of the overhead factor

The authors compared their prover with vLLM, an optimized inference-serving engine.

The comparison is expressed in GPU time, rather than simply elapsed wall-clock time.

For illustration, suppose ordinary inference requires one GPU-second for a particular workload.

An overhead of 70–95× would imply:

Operation

Illustrative GPU time

Ordinary inference

1 GPU-second

Cryptographically proven computation

70–95 GPU-seconds

These values illustrate the ratio; they are not actual timings for an individual FlashProve request.

The original experiments used the 4-bit integer model for FlashProve and an FP8 implementation of the same model for vLLM, running on eight H100 GPUs.

The authors normalized the comparison by GPU utilization. Therefore, the result is informative about relative GPU resource consumption, but it is not a perfectly identical numerical-workload comparison.

The comparison also focuses on prefill processing, not complete end-to-end conversational response generation.


The overhead is substantial. It means that providing cryptographic assurance for every inference request could require much more accelerator capacity than ordinary model serving.

However, it is also a major research achievement, because previous open-source approaches to proving very large neural-network computations were much slower or produced much larger proof files.

8. What does this achievement actually prove?

There are several important qualifications.

First, the proof verifies model-computation integrity, not factual accuracy. A correctly verified model can still generate incorrect, misleading or biased information.

Second, the proof is tied to a committed model configuration. It does not, by itself, establish that the committed model is safe, well-designed or suitable for the requested task.

Third, FlashProve's primary demonstration concerns a quantized integer model using greedy token selection. Its results are not automatically transferable to arbitrary floating-point inference or stochastic sampling.

Fourth, the supported context window in the published prototype is only 1,024 tokens—much shorter than the context windows used by many contemporary AI applications.

Finally, FlashProve is a research prototype. Its software implementation and formal verification framework still have limitations that the authors explicitly acknowledge.

9. Implications for AI security and digital trust

The broader significance is that cryptographic verification could introduce a new trust mechanism into AI infrastructure.

Consider a company that pays an API provider for responses from a particular advanced AI model.

Without inference verification, the client generally depends on the provider's assurance that the requested model was used.

With an appropriate proof system, the client could independently check whether the claimed result corresponds to the model weights previously committed by the provider.

This could be important in environments such as financial services, telecommunications security, cybersecurity auditing and regulated AI services.

The key distinction is between trust based on provider declarations and trust supported by independently verifiable cryptographic evidence.

Final interpretation

The paragraph can be summarized technically as follows:

FlashProve demonstrates that it is feasible to generate compact, independently verifiable cryptographic proofs for computations performed by an extremely large MoE language model. Using one H100 GPU supported by substantial CPU and RAM resources, the system processes approximately 110 prompt-token positions per second in a batched proving workload and produces proofs of approximately 19 MB per request.

The main limitation is the cost: generating these proofs requires roughly 70–95 times the GPU time of the measured ordinary prefill workload.

The most significant result is therefore not faster AI inference. It is the ability to establish cryptographic assurance about which committed model performed a computation, while keeping its parameters confidential and allowing verification without rerunning that model.

Comments

Popular posts from this blog

IBM goes big to keep quantum computers cold

IBM - avances en mantener adecuadamente fríos computadores cuánticos.

Estructura multinivel de los sistemas de agentes de IA