Jul 15, 11:32 AM

Meta and DeepMind Develop Efficient AI Models with Mixture-of-Experts Architecture Leading AlpacaEval 2.0 with 65.1%

Meta is focusing on developing AI models for mobile devices, leveraging the Mixture-of-Experts (MoE) architecture. This approach, highlighted in the 'Mixture of a Million Experts' paper, introduces a technique called Parameter Efficient Expert Retrieval (PEER) that enables the creation of MoE models with over a million small experts. This method remains computationally efficient and offers a better performance-compute tradeoff compared to traditional models. DeepMind's PEER scales language models with millions of tiny experts, reducing inference cost and memory usage, and overcoming challenges like catastrophic forgetting. Meta researchers, along with Databricks' MosaicAI team, are exploring tools such as MegaBlocks within the PyTorch framework to facilitate MoE development. The MoA model using open-source LLMs leads AlpacaEval 2.0 with a score of 65.1%.

#Meta #MoE #Parameter Efficient Expert Retrieval #DeepMind #Databricks #MosaicAI #MegaBlocks #PyTorch #MoA #AlpacaEval

Written with ChatGPT (GPT-4o).