Oct 11, 10:08 AM

NVIDIA's New Method Enhances Large Language Models with 1.5% Better Loss

NVIDIA has introduced a new method for enhancing large language models by upcycling them into Mixture of Experts (MoE). This approach, detailed in their latest research, involves upcycling the Nemotron-4 15B model on 1 trillion tokens. The findings indicate that upcycling outperforms continuous dense model training, achieving a 1.5% better loss in both coarse and fine MoE models. The study also introduced a virtual groups initialization scheme. This research highlights the potential for increased efficiency and effectiveness in large-scale language model training.

#NVIDIA #Mixture of Experts #Nemotron

Written with ChatGPT (GPT-4o).

Sources

fly51fly@fly51fly
1 year ago
[CL] Upcycling Large Language Models into Mixture of Experts E He, A Khattar, R Prenger, V Korthikanti... [NVIDIA] (2024) https://t.co/Hmj6lYFWoy https://t.co/adWSqww5zU
Deep Learning Weekly@dl_weekly
1 year ago
🤖 From this week's issue: A visual guide exploring the concept of Mixture of Experts (MoE) and its application in large language models. https://t.co/7n5SxLXG7x
Aran Komatsuzaki@arankomatsuzaki
1 year ago
NVIDIA presents Upcycling Large Language Models into Mixture of Experts Finds that upcycling outperforms continued dense model training based on large-scale experiments using Nemotron-4 15B trained on 1T tokens https://t.co/lKEtbMeQX8 https://t.co/L4LiEKrWDm

Additional media

Image #1 for story nvidia-s-new-method-enhances-large-language-models-1-5-better-loss-e356e32b

Image #2 for story nvidia-s-new-method-enhances-large-language-models-1-5-better-loss-e356e32b

Image #3 for story nvidia-s-new-method-enhances-large-language-models-1-5-better-loss-e356e32b

Image #4 for story nvidia-s-new-method-enhances-large-language-models-1-5-better-loss-e356e32b

Image #5 for story nvidia-s-new-method-enhances-large-language-models-1-5-better-loss-e356e32b

Image #6 for story nvidia-s-new-method-enhances-large-language-models-1-5-better-loss-e356e32b