NVIDIA has introduced a new method for enhancing large language models by upcycling them into Mixture of Experts (MoE). This approach, detailed in their latest research, involves upcycling the Nemotron-4 15B model on 1 trillion tokens. The findings indicate that upcycling outperforms continuous dense model training, achieving a 1.5% better loss in both coarse and fine MoE models. The study also introduced a virtual groups initialization scheme. This research highlights the potential for increased efficiency and effectiveness in large-scale language model training.