Sources
clem 🤗Just $10M and two months to train from scratch a GPT3.5 - Llama2 level model. For context, it probably cost 10-20x more to OAI just a year ago! The more we improve as a field thanks to open-source, the cheaper & more efficient it gets! All companies should now train their own… https://t.co/3xnoNwt03p
clem 🤗That's the exact opposite IMO! $10M to train a GPT3.5 level model whereas it probably cost OAI at least 10-20x more just a year or two ago. The more we improve as a field thanks to open-source, the cheaper & more efficient it gets to produce the same capabilities. Let's go… https://t.co/bEtUXvVb4n
Bindu ReddyHuge models that barely beat GPT 3.5 aren't as interesting as 7B models that can beat GPT 3.5 or Claude Haiku, which offers dirt-cheap inference and GPT-4 perf. Fine-tuning is also best done on smol models.

