Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimation

Wed, 25 Feb 2026 00:00:00 +0000

The Core Problem

The introduction of the NVFP4 lower-precision format (natively supported by NVIDIA Blackwell GPUs) opens the door for end-to-end fully-quantized pre-training of massive Large Language Models (LLMs). However, existing quantized training methods compromise the format’s representation capacity. Specifically, to achieve unbiased gradient estimation via stochastic rounding (SR), these methods sacrifice noticeable accuracy compared to standard FP16 and FP8 training.

Key Observations

To solve this, the authors introduce MS-EDEN, a novel unbiased quantization routine specifically designed for micro-scaled formats.

Reduced Error: MS-EDEN provides more than 2x lower quantization error than traditional stochastic rounding (SR).
Quartet II Integration: MS-EDEN is seamlessly integrated into a new fully-NVFP4 quantization scheme for linear layers, aptly named Quartet II.

Analytically, Quartet II achieves consistently better gradient estimation across all major matrix multiplications—in both forward and backward passes. Furthermore, the proposal pairs exceptionally well with other recent performance enhancements aimed at NVFP4.

Results

The authors validated Quartet II on end-to-end LLM training runs with up to 1.9 Billion parameters on 38 Billion tokens.

Supplied kernels for execution on NVIDIA Blackwell GPUs demonstrate an impressive up to 4.2x speedup over equivalent BF16 implementations.

Quartet: Native FP4 Training Can Be Optimal for Large Language Models

Wed, 25 Feb 2026 00:00:00 +0000

The Low-Precision Training Challenge

When training large language models (LLMs), moving to low-precision offers a compelling way to improve throughput and energy efficiency. NVIDIA’s recent Blackwell architecture even provides hardware support for very low-precision operations using FP4 variants.

However, existing algorithms for training in FP4 face severe accuracy degradation. To compensate, they often fallback to mixed-precision, effectively leaving performance on the table.

Key Observations

Instead of mixed-precision workarounds, we investigate end-to-end, native FP4 training. Through extensive evaluations on Llama-type models, we explored how to keep all major computations (linear layers) in pure low precision.

We reveal a new low-precision scaling law that precisely quantifies the performance trade-offs across different bit-widths and training setups.

Introducing Quartet

Guided by this scaling law, we formulated Quartet — an optimal technique for balancing accuracy and computation in FP4.

We implemented Quartet using custom, optimized CUDA kernels tailored for the Blackwell architecture. What we discover is that fully FP4-based training can be a highly competitive alternative to both FP16 half-precision and FP8 training, closing the accuracy gap without sacrificing speed.

Read the Paper →
Code and Implementation →

Disclaimer: This blog post was automatically generated from the arXiv paper.

LLM | Andrei Panferov

Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimation

The Core Problem

Key Observations

Results

Links

Quartet: Native FP4 Training Can Be Optimal for Large Language Models

The Low-Precision Training Challenge

Key Observations

Introducing Quartet