Aug 28, 03:20 PM

Mamba in the Llama: New Study on RNN and Transformer Hybrid Models

Researchers Junxiong Wang, Daniele Paliotta, Avner May, Alexander M. Rush, and Tri Dao have released a new study titled "The Mamba in the Llama: Distilling and Accelerating Hybrid Models." This work explores the potential of distilling large Transformer models into linear Recurrent Neural Networks (RNNs) by reusing linear projection weights from attention layers. The study highlights the competitive performance of linear RNN architectures, such as Mamba, in language modeling compared to Transformer models, while also emphasizing their advantageous deployment characteristics. The research is a collaborative effort among experts from Cornell University, the University of Geneva, and Together AI.