May 13, 04:15 PM

Hugging Face Deploys Whisper Large V3 for 8x Faster Transcriptions at $0.80/hr on NVIDIA GPUs; PortkeyAI Optimizes Gateway Latency

Hugging Face has introduced a new deployment of OpenAI's Whisper speech recognition model on its Inference Endpoints platform, achieving transcription speeds up to eight times faster than previous versions. This enhancement leverages the vLLM project for optimized inference on NVIDIA GPUs such as the Ada Lovelace L4 and L40s, incorporating PyTorch compilation techniques. The upgraded Whisper Large V3 model transcribes audio at 100 times real-time speed on a $0.80 per hour L4 GPU, enabling the transcription of one hour of audio for less than one cent. These improvements significantly reduce latency and cost without requiring any configuration changes from users. Additionally, PortkeyAI has rolled out optimizations on its Gateway that also reduce latency for OpenAI embeddings, further enhancing response times.

#Hugging Face #OpenAI #Whisper #Inference Endpoints #NVIDIA #Ada Lovelace L4 #PyTorch #Whisper Large V3 #PortkeyAI #Gateway

Written with ChatGPT (GPT-4).

Sources

Additional media

Image #1 for story hugging-face-deploys-whisper-large-v3-8x-faster-transcriptions-0-80-hr-on-nvidia-85d8d3cd

Image #2 for story hugging-face-deploys-whisper-large-v3-8x-faster-transcriptions-0-80-hr-on-nvidia-85d8d3cd

Image #3 for story hugging-face-deploys-whisper-large-v3-8x-faster-transcriptions-0-80-hr-on-nvidia-85d8d3cd