Groq's Mixtral Delivers 483-500 Tok/s, Undercuts GPT 3.5 at $0.8/1M Tokens

Groq Inc., a tech company, is making headlines with its Mixtral technology, delivering an impressive processing speed of nearly 483 to 500 tokens per second (tok/s). This breakthrough in speed is significantly impacting the user experience (UX) for Large Language Models (LLMs), offering instantaneous responses and enabling new use-cases. The technology's affordability is also noteworthy, with costs reported at 27 cents per 1 million tokens, making it cheaper than GPT 3.5, and another pricing point mentioned at $0.8 per 1 million tokens. Founded by former Google TPU members, Groq's innovation is seen as a potential game-changer in AI development, challenging the dominance of traditional GPUs and opening up possibilities for real-time conversations with AI models. The company's public demo showcased an AI Answers Engine capable of generating factual, cited answers in less than a second. Industry observers are excited about the implications for performance and UX, highlighting Groq's role in overcoming previous bottlenecks in cost and latency in LLMs. Additionally, Google Gemini Ultra, a closed-source technology, can handle 500k tokens, contrasting with Groq's open approach.