MistralAI Launches 8x22B MoE Model with 65k Context, Shakes Up AI Landscape

MistralAI has recently released a new 8x22B Mixture of Experts (MoE) model, sparking discussions and analyses within the tech community. The model, noted for its significant size and potential capabilities, features a 65k context size, and comparisons are already being drawn to its ability to potentially outperform existing models such as DBRX and even ChatGPT-3.5, though it may still fall short of more advanced models. Technical details shared by enthusiasts reveal that the tokenizer used in this model is identical to that of Mistral's previous 7b model, and the base model size has increased. The new model's hardware requirements are substantial, with a 16bit precision needing 258GB of VRAM, though optimizations could reduce this to 73GB or even 58GB with specific configurations. The release has also raised questions about MistralAI's product lineup and strategy, particularly in how this model fits into it and what it signifies for future developments.