Jul 16, 04:06 AM

H2O.ai Releases H2O-Danube3 Language Models Under Apache 2.0, Trained on 6T and 4T Tokens

H2O.ai has released its latest series of small language models, H2O-Danube3, under the Apache 2.0 license. The series includes two models: H2O-Danube3-4B, which is trained on 6 trillion tokens, and H2O-Danube3-500M, trained on 4 trillion tokens. These models were pre-trained on high-quality web data in three stages and utilize the Llama architecture with a Mistral tokenizer (32K vocabulary). The models feature an 8192 context length and Grouped Query Attention. H2O-Danube3 models show competitive performance on various benchmarks, including academic and chat tasks, and outperform Qwen 2 0.5B while being competitive with Phi3 4B.

#Apache #Llama #Mistral #Grouped Query Attention #Qwen #Phi3

Written with ChatGPT (GPT-4o).