Apr 15, 04:05 PM

EleutherAI Launches Pile-T5, Trained on 2T Tokens, Enhances AI Benchmarks

EleutherAI has released a new T5 model, named Pile-T5, which has been trained on 2 trillion tokens from the Pile using the Llama tokenizer. This model, featuring intermediate checkpoints, offers a significant boost in benchmark performance. The development was led by Aran Komatsuzaki and Lintang Sutawika, under the guidance of Colin Raffel. The project, described as a long labor of love, also benefited from the computational resources contributed by EMostaque. Additionally, the model is noted for its reproducibility and its potential utility in both natural language and code applications. This fully open release incorporates FLAN and utilizes permissible commercially licensed datasets provided by TeraflopAI.

#EleutherAI #Pile #Llama #Aran Komatsuzaki #Lintang Sutawika #Colin Raffel #EMostaque #TeraflopAI

Written with ChatGPT (GPT-4).

Sources

Enrico Shippole@EnricoShippole
3 years ago
Data is what makes the model. We at @TeraflopAI are working hard to provide the open-source community with permissible commercially licensed datasets for training. Congrats to @arankomatsuzaki, @lintangsutawika, and @colinraffel. And thanks to @ShayneRedford for his work on FLAN. https://t.co/DheOvHTeil
TeraflopAI@TeraflopAI
3 years ago
Glad to see our very own @arankomatsuzaki pushing the boundaries of open-source research with a new T5 release using our data. Congrats to @lintangsutawika and @colinraffel. And @ShayneRedford for his great efforts on FLAN. https://t.co/0oZeOZhZhs
Emad@EMostaque
3 years ago
A long labor of love by the team for a new high quality T5 model. Happy to have contributed compute resources for this, will be useful for research & more as a fully open release 🚀 https://t.co/pEoshSkPWq

EleutherAI Launches Pile-T5, Trained on 2T Tokens, Enhances AI Benchmarks

Sources

Additional media

Similar Stories