LlaMa-Omni Launches with 226ms Latency; Ichigo-Llama 3.1 Trained on 50,000 Hours of Speech

A new model named LlaMa-Omni has been introduced, designed to rival GPT-4o for real-time speech interaction with large language models (LLMs). This model can generate both text and speech from speech instructions, achieving a response latency as low as 226 milliseconds. Additionally, the Ichigo-Llama 3.1, a local real-time voice AI, has been unveiled with two significant enhancements: it can now respond verbally and recognize when it cannot comprehend input. The Ichigo model is capable of running on a single NVIDIA 3090 GPU and is open source, featuring the latest llama3-s checkpoint and trained on 50,000 hours of speech across seven languages. The model utilizes an early-fusion audio and text multimodal approach and has been instruction-tuned on 1.89 million samples.