Study Using Cosmopedia v2 Shows Hybrid Data Outperforms Pure Synthetic Data in GPT-2 Training

Recent experiments in machine learning have highlighted the potential benefits and limitations of using synthetic data for training language models. Yuchenj_UW conducted a study using GPT-2 (350M) models, revealing that while synthetic data can initially boost performance, it tends to underperform compared to hybrid data approaches. The study involved training four GPT-2 models with synthetic data for 100 billion tokens using Cosmopedia v2. The findings suggest that hybrid data, which combines synthetic and real data, results in stronger models. This aligns with observations that synthetic data alone may lead to repetitive patterns, which could hinder model performance. The debate continues on the efficacy of synthetic data, with some experts noting that the difference between the best and worst configurations is minimal.