Nemotron Labs TwoTower 30B A3B

Nemotron-Labs-TwoTower-30B-A3B-Base-BF16 is a block-wise autoregressive diffusion language model that generates text by iteratively denoising blocks of tokens in parallel rather than one token at a time. It is built on the NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16 backbone and uses two towers: a frozen causal autoregressive context tower that processes the prompt and committed tokens, and a trainable diffusion/denoiser tower that generates blocks of tokens via mask diffusion with bidirectional in-block attention, layer-aligned cross-attention to the context tower, and context-seeded Mamba states. Both towers are copies of the same 52-layer hybrid Mamba-2/attention/MoE backbone; only the denoiser tower is trained.

Key Capabilities and Architecture

• Generation modes: Mask diffusion (block-wise iterative denoising with confidence-based unmasking), Mock-AR (two-tower autoregressive), and standard AR (single tower).
• Parameters: ~60B total (30B per tower), with ~3B active parameters per token via 128 routable experts (6 activated plus 2 shared).
• Training: The denoiser was trained on ~2.1T tokens using a masked-diffusion objective, with the backbone pretrained on 25T tokens. Precision: BF16. Software: Megatron-LM.
• Context length: Up to 128K tokens.

Benchmark Performance

At the default operating point (confidence threshold γ=0.8, block size S=16, BF16 on 2×H100 GPUs), the model retains 98.7% of the autoregressive baseline's aggregate benchmark quality and achieves 2.42× the baseline's wall-clock generation throughput.

Task	AR Baseline	TwoTower (Diffusion)
MMLU (5-shot, acc)	78.56	78.24
MMLU-Pro (5-shot, CoT EM)	62.59	60.93
ARC-Challenge (25-shot, acc_norm)	91.72	92.66
WinoGrande (5-shot, acc)	76.09	76.09
RACE (0-shot, acc)	88.90	88.90
HumanEval (0-shot)	79.27	75.58
MBPP-Sanitized (3-shot)	74.71	74.28
GSM8K (8-shot, acc)	92.49	90.14
MATH-500 (4-shot)	84.40	80.60
MMLU Global Lite (5-shot, avg acc)	73.97	73.94
MGSM (8-shot, avg acc)	80.80	80.40

How Mask Diffusion Works

Generation is block-wise autoregressive: the context tower encodes the prompt once, then the denoiser fills one block of block_size positions at a time. For each new block, it initializes all positions as [MASK], then for steps_per_block iterations computes the diffusion timestep, runs the denoiser with bidirectional in-block attention and cross-attention to the context cache, and commits high-confidence positions. Multiple tokens may be committed per step.

The model is governed by the NVIDIA Nemotron Open Model License Agreement. Developed by NVIDIA Corporation (September 2025 – April 2026), with pre-training data cutoff June 25, 2025.