UnslothAI and NeuralMagic Optimize Mistral v3 for 2x Faster Inference and Longer Context Support

Check this out! Only a few hours after its release, our team optimized Mistral-7B-Instruct-v0.3 for 3x faster deployment with #vLLM. You can now fit the entire model and full 32k context length inside a single A10 GPU. 🙌👏 https://t.co/RzxJpVC6Vl

Check this out! Only a few hours after its release, our team optimized Mistral-7B-Instruct-v0.3 for 3x faster deployment. You can now fit the entire model and full 32k context length inside a single A10 GPU. 🙌👏 https://t.co/RzxJpVC6Vl

Awni Hannun@awnihannun

2 years ago

Checkout the new Mistral v0.3 models with MLX LM. Pre-quantized models in the 🤗 MLX community https://t.co/dUgErUXnM3 h/t @Prince_Canuma ! Generating 512 tokens at 107 toks/sec with the 4-bit model on an M2 Ultra. Models got better but just as fast as ever: https://t.co/NHJ50x22X0