Jul 20, 12:15 AM

EfficientQAT and Q-Sparse Boost Efficiency in Large Language Models

Recent advancements in artificial intelligence have led to significant improvements in the efficiency and performance of large language models (LLMs). Efficient Quantization-Aware Training (EfficientQAT) has introduced a new algorithm that allows a 2-bit INT Llama-2-70B model to outperform the FP Llama-2-13B model while using less memory. This method achieves performance on par with AQLM but is 10 times faster in quantization. EfficientQAT can obtain a 2-bit Llama-2-70B model on a single A100-80GB GPU in 41 hours. Additionally, researchers from Microsoft and the University of Chinese Academy of Sciences have developed Q-Sparse, an approach for training sparsely-activated LLMs, enabling full sparsity of activations. XTuner's Zero Memory Waste and Sequence Parallel in Preference Alignment have also been highlighted, cutting DPO training time in half and allowing Llama3 70B RM to train with sequence lengths up to 1 million tokens on 64 A100s, achieving 50% faster training.

#Microsoft

Written with ChatGPT (GPT-4o).

Sources

Additional media

Image #1 for story efficientqat-q-sparse-boost-efficiency-large-language-models

EfficientQAT and Q-Sparse Boost Efficiency in Large Language Models

Sources

Additional media

Similar Stories