Jul 11, 06:26 PM

OpenAI's GPT-6 to Have 90x Compute of GPT-4, Generating 1.2 Quadrillion Words Annually

OpenAI's advancements in large language models (LLMs) continue to make significant strides. GPT-6 is projected to have approximately 90 times the computational power of GPT-4. The cost of training models has dramatically decreased, with GPT-2 class models now trainable for less than $1,000. For instance, training a GPT-2 model today costs around $672 on an 8XH100 GPU node for 24 hours. This trend is expected to continue, with smaller models becoming more efficient and LLM inference becoming ubiquitous. OpenAI is generating about 1.2 quadrillion words annually, highlighting the massive scale of its operations. For context, Llama 3 was trained on 15 trillion tokens, which is about 120 times fewer words.

Written with ChatGPT (GPT-4o).

Sources

renji the synthetic data maximalist@brickroad7
2 years ago
OpenAI generating about 1.2 quadrillion words every year. (🤯) (For context, Llama 3 was trained on 15 trillion tokens. About 120x less words.) https://t.co/t15EyQq2k2
Bindu Reddy@bindureddy
2 years ago
Today GPT-2 class models can be trained for < $1K 🤯 We are going to see the following trends in the next 5 - Smaller models will become performant - LLMs become cheaper and cheaper train - LLM inference becomes ubiquitous We should expect to see several Sonnet 3.5 class…
Andrej Karpathy@karpathy
2 years ago
In 2019, OpenAI announced GPT-2 with this post: https://t.co/tZzc5DoXEd Today (~5 years later) you can train your own for ~$672, running on one 8XH100 GPU node for 24 hours. Our latest llm.c post gives the walkthrough in some detail: https://t.co/dUnbh1LshJ Incredibly, the… https://t.co/ZI9qfpuLse

OpenAI's GPT-6 to Have 90x Compute of GPT-4, Generating 1.2 Quadrillion Words Annually

Sources

Additional media

Similar Stories