Nov 15, 05:14 PM

Groq's New Llama 3.1 70B Endpoint Achieves 1,665 Tokens/s with Speculative Decoding

Groq has launched a new Llama 3.1 70B endpoint, achieving over 6 times faster output tokens per second compared to their current endpoint and over 20 times the median of other providers. The new endpoint achieves 1,665 output tokens per second by leveraging speculative decoding. Groq's 14nm V1 LPU is currently delivering great performance and price for inference, with a 4nm V2 expected next year. Additionally, GroqCloud has made Llama-3.1-70B-specdec available to play with for everyone on Dev Tier.

#Groq #GroqCloud #Dev Tier

Written with ChatGPT (GPT-4o).

Sources

Benjamin Klieger@BenjaminKlieger
2 years ago
250 T/s —> 1665 T/s 😎 Any app powered by Groq with 70b spec decoding would now run ~6x faster⚡️, the power of Groq speed! https://t.co/Z2PtRahvMH
Artificial Analysis@ArtificialAnlys
2 years ago
Groq has launched a new Llama 3.1 70B endpoint with >6X faster output tokens/s than their current endpoint and >20X the median of other providers @GroqInc's new endpoint achieves 1,665 output tokens/s through leveraging speculative decoding. Speculative decoding is an inference… https://t.co/eAs9OHIK6Z
Hatice Ozen@ozenhati
2 years ago
Friday Fun: You wanted even faster inference for your apps, so we just dropped llama-3.1-70B-specdec on GroqCloud - now available to play with for everyone on Dev Tier. 🏁 https://t.co/KAYt2LXnrE

Additional media

Image #1 for story groq-s-new-llama-3-1-70b-endpoint-achieves-1665-tokens-s-speculative-decoding-78e76d11

Groq's New Llama 3.1 70B Endpoint Achieves 1,665 Tokens/s with Speculative Decoding

Sources

Additional media

Similar Stories