Google Releases DiffusionGemma: When Language Models No Longer Write Only Left to Right

Google recently released the open-source model DiffusionGemma, alongside the DiffusionGemma technical report. Starting directly from the fully post-trained weights of Gemma 4 26B-A4B, it uses a discrete multinomial diffusion algorithm to transform the conventional autoregressive generation process—where tokens are committed unidirectionally one by one—into a parallel generation mechanism that iteratively denoises across multiple rounds of bidirectional attention within a 256-token canvas.

Compared with earlier industry efforts that were closed-source or left model parameters and training details partially undisclosed, Google has released the Google model card while directly open-sourcing the full model parameters on the Hugging Face open weights page. The architecture also retains an AR fallback interface to switch back to classic autoregressive mode, allowing engineering teams to deploy and evaluate discrete text diffusion directly on mainstream inference engines.

In the officially published figures, three numbers create a striking tension: on a single NVIDIA H100 GPU, the model achieves a single-request decode speed of 1,456 tokens per second; the supplemental training token budget is less than 10% of the original Gemma 4 total training budget; yet on the AIME 2026 mathematical reasoning benchmark, the score falls from the original model’s 88.3 to 69.1. As an engineer tracking the evolution of language model inference, my first instinct upon seeing the release was naturally drawn to the four-digit decoding speed. But after cross-checking the test conditions and ablation data, the more central question emerged: what does this architectural experiment—trading generation order for inference acceleration—actually change in underlying execution, and what price does it extract in model capabilities?

Once Written, a Prefix Becomes History—but Inside a Local Canvas, It Can Be Revised Repeatedly

Standard autoregressive language models generate text based on the conditional probability $p(x_i | x_{<i})$, predicting the next token from left to right, one token at a time. Each forward pass builds strictly upon the existing prefix. Take multi-digit multiplication 23 × 47 = 1081 as an example: if the four digits are treated as four generation positions, once the autoregressive model outputs 1 at the first step, that position is immediately frozen into historical prefix context that subsequent computations must accept. If the model outputs an incorrect digit in that first step, within the same generation trajectory, subsequent computation cannot return to the first position to modify it directly; it can only acknowledge the mistake in later text or rely on an external harness to retry.

Text diffusion changes this generation process. It starts by laying out multiple unfinalized candidate positions, and then evolves the entire block of text simultaneously across multiple forward passes. In the first round of computation, the second digit might temporarily be 1, resulting in an interim state of 1181; in the next forward pass, the model uses bidirectional attention to reference information from both preceding and succeeding positions, correcting the second digit to 0 under contextual constraints. As denoising progresses, high-confidence positions lock in first, while low-confidence positions continue to be adjusted until the entire text stabilizes.

DiffusionGemma confines this bidirectional iteration to a fixed local window. The model processes a canvas region containing 256 tokens at a time. Within the current canvas, the model repeatedly adjusts all candidate tokens across multiple forward passes. Once the current canvas completes denoising and is finalized, the model writes it fully into the KV cache, serving as read-only prefix context for subsequent generation steps. When the model processes a new canvas block, it cannot write back to finalized historical blocks:

prompt / 已提交块 1 / 已提交块 2 / 正在修改块 3
           只读             只读          块内可改

In actual generation, research on DiffusionGemma token commitment dynamics recorded 686 test prompts and found that the model does not finalize all 256 positions inside the canvas at the very last step in one shot; instead, it exhibits a dynamic, batched locking process that varies by task and text granularity. This mechanism breaks the autoregressive constraint where every token must be committed immediately, while preserving a block-level forward causal structure. A single forward pass can advance multiple positions simultaneously, but the model can only modify content within the active 256-token canvas.

Text Cannot Directly Take Gaussian Noise, but Existing Language Models Can Adopt an Alternative Generation Mode

Diffusion models have long matured in the image domain, but transferring them to natural language processing required years of exploration. Image pixels and latent space features exist in continuous numerical spaces; when perturbed slightly or overlaid with Gaussian noise, they remain on continuous manifolds, enabling neural networks to learn step-by-step denoising via continuous score functions. Text token IDs, by contrast, are discrete categorical indices: adding continuous values to a specific token ID is undefined in a vocabulary, and adjacent indices lack continuous semantic relationships. Because text cannot directly receive Gaussian noise, specialized noising and denoising rules must be formulated for discrete spaces.

The mathematical formalization of discrete noising saw key progress in 2021. The D3PM paper introduced structured transition matrices and absorbing state mechanisms, solving the transition probability modeling problem for how tokens are progressively noised and denoised in discrete spaces. Another exploratory path mapped tokens into continuous word embedding spaces to apply Gaussian diffusion; the 2022 Diffusion-LM paper demonstrated the flexibility of continuous trajectories in controllable generation and local infilling, but the rounding errors introduced when mapping continuous vectors back to discrete tokens degraded text quality, and its 200-step denoising sampling resulted in end-to-end latency 7 times that of an autoregressive baseline.

Subsequently, discrete diffusion progressively closed the modeling quality gap with autoregressive models. The SEDD paper, published in 2023, reduced perplexity by 50% to 75% relative to earlier discrete diffusion baselines on the LM1B dataset with GPT-2-small scale experiments, narrowing the perplexity gap to within roughly 1 PPL of a scratch-trained autoregressive baseline under the same configuration.

In the large language model era, the compute cost of training from scratch prompted researchers to pivot toward repurposing mature weights. The LLaDA paper trained an 8B discrete diffusion model from scratch using 2.3T tokens, demonstrating that discrete diffusion can also develop in-context learning and instruction-following capabilities at scale, while highlighting the massive cost of pretraining large models from zero. In response, the DiffuLLaMA paper explored a direct conversion route, converting pretrained GPT-2 and LLaMA2 checkpoints (127M to 7B parameters) into diffusion models; on the 7B model, adaptation was completed using roughly 60B to 65B tokens—well below the study’s 200B token ceiling—proving that diffusion models do not need to relearn general language distributions from scratch.

Engineering deployment and inference latency subsequently became the new focus. The Mercury Coder technical report showcased a low-latency diffusion model designed for code scenarios; in Artificial Analysis evaluations using specialized code prompts with roughly 1K input tokens paired with 1K output tokens, it achieved decode speeds of 1,109 and 737 output tok/s on a single H100, though its parameter count, initialization source, and specific training recipe were not disclosed. The Gemini Diffusion official page published a sampling speed of 1,479 tokens per second (excluding a 0.84-second fixed overhead); without disclosing the hardware specifications or context length, its evaluations showed substantial performance gaps compared to Gemini 2.0 Flash-Lite on benchmarks such as GPQA (40.4 vs. 56.5) and Global MMLU (69.1 vs. 79.0). Following this, the Dream paper and LLaDA 2.0 paper pushed further forward on model conversion, MoE architectures, chunk-based diffusion, and KV cache integration.

DiffusionGemma continues the established trajectory of text diffusion and weight conversion. Its technical delta lies in integrating mature model asset reuse, few-step sampler distillation, autoregressive fallback, open weights, and mainstream inference engine support into a single, directly distributable engineering artifact. The model takes the fully post-trained Gemma 4 26B-A4B weights as its initialization starting point. This base model possesses 25.2B total parameters, activating roughly 3.8B parameters per token, and already provides instruction following, long thinking, multimodal understanding, and tool calling capabilities out of the box.

For the noising process, the model employs multinomial discrete diffusion, replacing tokens with random tokens uniformly sampled from the vocabulary at probabilities conditioned on the noise level. The training process consists of two stages: the model first learns during denoising adaptation training how to reconstruct clean text from a 256-token canvas corrupted with random tokens. After this step, the model can generate normal text when using a relatively large number of sampling steps, but compressing sampling to very few steps causes quality collapse. Next, Sampler Distillation and Reinforcement Learning (SD·RL) unifies task reward maximization and sampling step compression into a single objective; with trajectory guidance provided by a high-step teacher model, the student model is trained to quickly converge on finalized text using fewer forward computation passes. Following SD·RL training, the model achieves an average of approximately 19.74 effective tokens per forward pass (TPF) across its entire output trajectory. This is an amortized average and does not imply that the model writes a fixed 19.74 tokens in every individual forward pass.

According to details disclosed in the DiffusionGemma technical report, the total token volume consumed during the supplemental training across both stages was less than 10% of the initial Gemma 4 model’s total training token budget. This proportion describes the training token budget and does not equate to the overall compute cost being only 10% of the original model. The report does not disclose the absolute number of supplemental training tokens consumed, FLOPs, GPU hours, or the complete conversion recipe.

Test Conditions Behind 1,456 Tokens per Second, and Latency Behavior Under Real Requests

The officially reported decode speed of 1,456 output tok/s corresponds to a precise set of hardware and workload test conditions.

According to the DiffusionGemma technical report, this figure was measured on a single NVIDIA H100 GPU, FP8 precision, single concurrent request (batch size 1), with 4096 input tokens paired with 1024 output tokens, and the measurement window covered only the decode phase, excluding prefill time. Under this configuration, the average hardware execution time for a single diffusion forward pass was 13.56 milliseconds, which, combined with the average throughput of 19.74 TPF, yields 1,456 output tokens per second. Under identical hardware and request settings, the standard Gemma 4 autoregressive model achieved a decode speed of 204 tok/s, or 303 tok/s with multi-token prediction (MTP) acceleration enabled.

This speedup stems from altering GPU memory access patterns. In single-request, low-concurrency scenarios, standard autoregressive models exhibit low compute intensity because each forward pass generates only 1 token, consuming much of the execution time in bandwidth overhead from loading model weights and the KV cache from VRAM into compute cores. DiffusionGemma processes 256 candidate tokens simultaneously in each forward pass, reducing the total number of sequential forward rounds and substantially slashing the redundant reads of model weights and the KV cache for a single request under low concurrency.

Regarding engine support, the vLLM official blog post published test results conducted jointly with Google and NVIDIA: in an FP8 environment at batch size 1, a single H100 recorded 1,008 generation tok/s, while an H200 reached 1,288 generation tok/s. This represents an engineered implementation resulting from targeted multi-party optimization, and should not be characterized as a completely independent, blind reproduction benchmark by third parties.

In production serving, peak TPS during the decode phase alone does not equate to user-perceived end-to-end response latency. The vLLM SPEED-Bench recipe provides a more comprehensive performance comparison on a single NVIDIA H100, BF16 precision, single concurrency, max_tokens=256, and using the recommended entropy-bounded sampler under a standardized evaluation suite:

These figures illustrate how throughput translates into practical interaction pipelines. While per-request generation TPS surged from 205 to 1282 (roughly a 6.2x improvement), the overall total output TPS only increased from 199 to 375 (roughly 1.9x) because DiffusionGemma generated shorter average text lengths than the autoregressive model on this test suite. The average end-to-end latency dropped from 2.87 seconds to 0.88 seconds, delivering an actual speedup of roughly 3.3x.

At the same time, average time to first token (TTFT) increased from 53 milliseconds for the autoregressive model to 489 milliseconds for the diffusion model. After finishing prefill, an autoregressive model requires only a single lightweight forward pass to emit its first token, whereas the diffusion model must complete multiple forward denoising iterations across the initial 256-token canvas until reaching commitment thresholds before it can begin streaming text output.

As concurrent requests increase, Google’s concurrency experiments show that at a test configuration of around 32 concurrent requests, total throughput in autoregressive mode begins to overtake diffusion mode. Under high concurrency, continuous batching aggregates multiple requests, saturating VRAM bandwidth and compute units while amortizing the single-step memory access overhead of autoregression across multiple requests. Consequently, the marginal benefit diffusion mode gains from reducing per-request memory access passes diminishes. This crossover point reflects only the test setup in the report, rather than a fixed system constant.

Once Speed Is Up, Why Does Complex Reasoning Drop Points—and How Much Can Switching Back to Autoregression Recover?

Alongside the dramatic surge in generation throughput, DiffusionGemma’s performance on complex reasoning tasks suffered a noticeable decline. The benchmark comparison in the Google model card records the benchmark scores for DiffusionGemma in diffusion thinking mode versus the original Gemma 4 in AR+MTP thinking mode:

On the math competition benchmark AIME 2026, the diffusion thinking mode score dropped from 88.3 to 69.1, a decline of 19.2 points; on the Tau2 benchmark, which evaluates multi-turn tool calling and agent planning, the score dropped from 68.2 to 56.2; on the long-context multi-needle retrieval task MRCR 128K, the score decreased from 44.1 to 32.0.

Standing in sharp contrast to this quality drop is the model’s retained AR fallback mechanism. When engineers reload DiffusionGemma’s final post-trained weights back into the original Gemma 4 classic autoregressive execution pipeline and execute the thinking process in autoregressive mode, its reasoning metrics rebound significantly: the AIME score recovers from 69.1 to 84.2 (compared to the original model’s 88.3), the GPQA score recovers from 73.2 to 79.8 (compared to the original model’s 82.3), and the BigBench Extra Hard score recovers from 47.6 to 59.1 (compared to the original model’s 64.8). Under this mode, the model’s decode speed drops back to the autoregressive baseline of approximately 204 tok/s.

The AR fallback test results provide a crucial attribution clue. The substantial metric recovery demonstrates that the two-stage supplemental training did not irreversibly damage the underlying knowledge representations stored in the weights, indicating that the runtime diffusion generation mode accounts for a portion of the quality degradation.

Yet this rebound does not equate to a lossless conversion. First, even after falling back to autoregressive mode, a performance deficit of 3 to 6 percentage points remains across metrics relative to the original Gemma 4. Second, because the technical report lacks comprehensive ablation studies isolating RL alone, distillation alone, and combined factor designs, it is impossible to cleanly quantify how much of the quality gap stems from data budget constraints during supplemental training, how much arises from distillation compressing sampling steps and limiting output diversity, and how much is caused by the shorter average output lengths induced by SD·RL training.

A mechanistic trade-off likely exists between high TPF generation and deep logical reasoning regarding the depth of serial dependencies. In autoregressive generation, for every reasoning token emitted, the model performs a directed conditional computation $p(x_i | x_{<i})$ conditioned strictly on the entire established prefix. Complex mathematical derivations and code logic possess strict serial causal dependencies, where subsequent deductive steps must build on the validity of preceding steps.

Under text diffusion mode, although a single forward pass can update roughly 20 tokens in parallel, these 20 tokens are adjusted collaboratively via bidirectional attention within the same forward step. This constitutes same-layer parallel updating, which mechanistically cannot equate to the causal computational depth of executing 20 consecutive sequential state transitions. The SD·RL training trajectory also reflects this tendency: once task rewards plateaued, continued training primarily suppressed the model’s prediction entropy and shortened output length; the final checkpoint’s average output length was nearly half that of the SFT checkpoint, a trade-off the authors acknowledged as sacrificing potential capability gains from long-chain reasoning.

Generation Order Is No Longer Singular, but Trade-offs Across Workloads Remain Distinct

Synthesizing execution mechanisms, throughput profiles, and reasoning capabilities, text diffusion’s position in engineering practice becomes clearer. It does not replace autoregression across the board; rather, it provides a runtime-configurable performance profile tailored to specific task characteristics.

Low-concurrency, long-output workloads represent a primary candidate profile for diffusion mode, as decode acceleration has ample room to offset higher time-to-first-token latency. However, existing SPEED-Bench results stem from a specific evaluation set and request profile, and cannot be extrapolated directly to all long-text tasks. Local text editing and infilling represent another prime experimental direction: bidirectional attention within the 256-token canvas can simultaneously consult context before and after the target region while allowing subsequent candidates to influence earlier positions. Nonetheless, DiffusionGemma lacks independent benchmarks for prose revision or cross-block editing, and once a canvas is committed to the KV cache, cross-block write-backs are impossible, meaning practical editing gains must still be validated in specific workflows.

Under higher concurrency, mature continuous batching already maximizes hardware utilization and aggregate throughput, diminishing the benefit diffusion mode derives from reducing per-request memory access. Latency-sensitive short tool calls and request routing typically emit only tens of tokens, where overall system latency is dominated by prefill and time to first token; here, diffusion mode’s elevated TTFT may actually increase response times. Strongly constrained structured output also continues to favor mature autoregressive stacks: autoregressive engines can enforce grammar or JSON Schema state machines token by token during generation, whereas discrete diffusion updates multiple positions in a single step, making it difficult for existing grammar state machines to apply identical token-level constraints directly. According to the discussion on structured outputs in the vLLM repository, diffusion decoding currently relies primarily on post-generation validation and retry.

Evaluating the future trajectory of this technical path hinges on upcoming empirical evidence. We need to observe whether diffusion models can sustain practical end-to-end latency advantages across a broader array of tasks when controlling for identical task accuracy and output length; whether native bidirectional canvas denoising can systematically surpass mature autoregressive models paired with external edit loops in code refactoring and prose editing quality; and, at the model architecture level, whether future sampler distillation algorithms or novel hybrid architectures can effectively bridge the gap in serial causal reasoning depth while maintaining high-TPF parallel output.

Until these questions are further validated by empirical data, the left-to-right unidirectional commitment paradigm of autoregression remains the default choice for complex logical reasoning and general production serving. What DiffusionGemma changes is that generation order ceases to be an immutable model assumption and becomes an operational choice that can be evaluated on a per-task basis.