paper-conference

Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimation
NVFP4 pre-training with MS-EDEN, an unbiased gradient estimator that reduces quantization error, and GPU kernels for NVIDIA Blackwell.
Andrei Panferov, Erik Schultheis, Soroush Tabesh, Dan Alistarh
PDF Cite Code Paper
Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization
A study of MXFP4 and NVFP4 inference, introducing Micro-Rotated-GPTQ and GPU kernels tailored to the constraints of microscaling formats.
Vage Egiazarian, Roberto Castro, Denis Kuznedelev, Andrei Panferov, Eldar Kurtic, Shubhra Pandit, Alexandre Marques, Mark Kurtz, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh
PDF Cite Code Paper arXiv
Apertus: Democratizing Open and Compliant LLMs for Global Language Environments
An open multilingual language model effort with transparent training recipes and a focus on data compliance and language coverage.
Alejandro Hernández-Cano, Alexander Hägele, Allen Hao Huang, Angelika Romanou, Antoni-Joan Solergibert, Barna Pásztor, Bettina Messmer, Dhia Garbaya, Eduard Frank Ďurech, Ido Hakimi, Juan Garcia Giraldo, Mete Ismayilzada, Negar Foroutan, Skander Moalla, Tiancheng Chen, Vinko Sabolčec, Yixuan Xu, Michael Aerni, Badr AlKhamissi, Inés Altemir Marinas, Mohammad Hossein Amani, Matin Ansaripour, Ilia Badanin, Harold Benoit, Emanuela Boros, Nicholas John Browning, Fabian Bösch, Maximilian Böther, Niklas Canova, Camille Challier, Clément Charmillot, Jonathan Coles, Jan Milan Deriu, Arnout Devos, Lukas Drescher, Daniil Dzenhaliou, Maud Ehrmann, Dongyang Fan, Simin Fan, Silin Gao, Miguel Gila, María Grandury, Diba Hashemi, Alexander Miserlis Hoyle, Jiaming Jiang, Mark Klein, Andrei Kucharavy, Anastasiia Kucherenko, Frederike Lübeck, Roman Machacek, Theofilos Ioannis Manitaras, Andreas Marfurt, Kyle Matoba, Simon Matrenok, Henrique Mendonça, Fawzi Roberto Mohamed, Syrielle Montariol, Luca Mouchel, Sven Najem-Meyer, Jingwei Ni, Gennaro Oliva, Matteo Pagliardini, Elia Palme, Andrei Panferov, Léo Paoletti, Marco Passerini, Ivan Pavlov, Auguste Poiroux, Kaustubh Ponkshe, Nathan Ranchin, Javier Rando, Mathieu Sauser, Jakhongir Saydaliev, Mukhammadali Sayfiddinov, Marian Schneider, Stefano Schuppli, Marco Scialanga, Andrei Semenov, Kumar Shridhar, Raghav Singhal, Anna Sotnikova, Alexander Sternfeld, Ayush Kumar Tarun, Paul Teiletche, Jannis Vamvas, Xiaozhe Yao, Hao Zhao, Alexander Ilic, Ana Klimovic, Andreas Krause, Çağlar Gu̇lçehre, David Rosenthal, Elliott Ash, Florian Tramèr, Joost VandeVondele, Livio Veraldi, Martin Rajman, Thomas C. Schulthess, Torsten Hoefler, Antoine Bosselut, Martin Jaggi, Imanol Schlag
PDF Cite DOI Paper arXiv
Quartet: Native FP4 Training Can Be Optimal for Large Language Models
End-to-end FP4 training of language models, combining low-precision scaling laws with CUDA kernels for Blackwell GPUs.
Roberto Castro, Andrei Panferov, Rush Tabesh, Oliver Sieberling, Jiale Chen, Mahdi Nikdan, Saleh Ashkboos, Dan Alistarh
PDF Cite Code Paper arXiv
QuEST: Stable Training of LLMs with 1-Bit Weights and Activations
Quantization-aware training with Hadamard normalization and a trust gradient estimator, enabling stable training down to 1-bit weights and activations.
Andrei Panferov, Jiale Chen, Soroush Tabesh, Mahdi Nikdan, Dan Alistarh
PDF Cite Code Paper arXiv
HIGGS: Pushing the Limits of Large Language Model Quantization via the Linearity Theorem
A theoretical link between layer reconstruction error and model perplexity, leading to data-free quantization and non-uniform bit allocation.
Vladimir Malinovskii, Andrei Panferov, Ivan Ilin, Han Guo, Peter Richtárik, Dan Alistarh
PDF Cite DOI Paper arXiv
Extreme Compression of Large Language Models via Additive Quantization
AQLM compresses language model weights through additive quantization, targeting accurate inference at very low bit widths.
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, Dan Alistarh
PDF Cite Code Paper arXiv