QuEST: Stable Training of LLMs with 1-Bit Weights and Activations

Abstract

Quantization-aware training with Hadamard normalization and a trust gradient estimator, enabling stable training down to 1-bit weights and activations.

Andrei Panferov

PhD Candidate in Computer Science

I work on efficient large language models, with a focus on low-precision training, quantization, and scaling laws.