Meta is focusing on developing AI models for mobile devices, leveraging the Mixture-of-Experts (MoE) architecture. This approach, highlighted in the 'Mixture of a Million Experts' paper, introduces a technique called Parameter Efficient Expert Retrieval (PEER) that enables the creation of MoE models with over a million small experts. This method remains computationally efficient and offers a better performance-compute tradeoff compared to traditional models. DeepMind's PEER scales language models with millions of tiny experts, reducing inference cost and memory usage, and overcoming challenges like catastrophic forgetting. Meta researchers, along with Databricks' MosaicAI team, are exploring tools such as MegaBlocks within the PyTorch framework to facilitate MoE development. The MoA model using open-source LLMs leads AlpacaEval 2.0 with a score of 65.1%.