Sources
Loading...
Additional media
Loading...
DeepNewz, mobile.
People-sourced. AI-powered. Unbiased News.
Developers are extending the context length of Llama 3 models, such as Llama-3-8B and Llama-3-70B, to improve performance. The context length has been increased from 8k to over 1.04M tokens using techniques like dataset augmentation and Rotary Position Embeddings. Meta's open-source approach allows for longer context windows, with models achieving up to 128K long-context support. Despite challenges in solving long-context issues, efforts are ongoing to enhance Llama-3 models' practical performance.
💬 At #FullyConnected2024, @Metas Product Director of GenAI @joespeez shares the real secret to Llama 3’s performance is in post-training. Get more exclusive industry insights at #FullyConnectedLondon. Register here → https://t.co/FfEt07XTdZ+ https://t.co/dkng2WkFQU
Context extension is cool if it works... many extended models do very very badly on any practical tasks. Even their performance on shorter context has been degraded. The only reliable method I'm finding (so far) for llama-3 is using the dynamic rope scaling up to 32k tokens (no…
Long Context On Llama-3 Is Yet To Be Solved Well We are doing some experiments on various models and realizing that we still have some ways to solve long-context correctly on Llama-3. Early experiments show pretty bad degradation on the MT bench score when the model is modified…