Apr 30, 10:45 PM

Llama-3 Models Hit 1M Context on Hugging Face, New Releases by InternLM

The AI community is abuzz with significant advancements in Llama-3 models, as various new developments have been announced. Notably, the Llama-3-8B model now supports an unprecedented 1 million context length, a substantial increase from the previous 160K context length, and is available on Hugging Face. This enhancement was made possible through the sponsorship of Crusoe Energy. Additionally, the Llama-3 model, trained on 15 trillion tokens, has shown sensitivity to quantization degradation due to its high precision in BF16. Furthermore, new fine-tuned models such as Llama-3-70B and Llamixtral-3 are being released, alongside the introduction of Vision Language Models based on Llama-3 8B and Phi-3 Mini by the InternLM team.

#Llama #Hugging Face #Crusoe Energy #Vision Language Models #InternLM

Written with ChatGPT (GPT-4).

Sources

AK@_akhaliq
2 years ago
Extending Llama-3's Context Ten-Fold Overnight We extend the context length of Llama-3-8B-Instruct from 8K to 80K via QLoRA fine-tuning. The entire training cycle is super efficient, which takes 8 hours on one 8xA800 (80G) GPU machine. The resulted model exhibits superior https://t.co/SZtBAm1F6b
Kirk Borne@KirkDBorne
2 years ago
💯🌟🚀@abacusai announces 128K long-context support for Llama-3-70B, putting it on par with #GPT4 regarding context length support and real-world usage. See post below from @bindureddy -and- check out the model here: https://t.co/mGxMv8uPz6 ⬅️❣️ https://t.co/wtlsVTKvhP
elvis@omarsar0
2 years ago
Nice to have 128K long-context support for Llama 3 70B already. It will be interesting to see how far these open models can get when extended with a bigger context. This seems to be an early release but curious to hear if anyone is building on top of these long-context open… https://t.co/Iz8OW7dyEv

Additional media

Image #1 for story llama-3-models-hit-1m-context-on-hugging-face-new-releases

Image #2 for story llama-3-models-hit-1m-context-on-hugging-face-new-releases

Image #3 for story llama-3-models-hit-1m-context-on-hugging-face-new-releases