Dec 12, 05:01 PM

New 'BIGGEN BENCH' Evaluates 103 Language Models Across 9 Capabilities; PortkeyAI Analyzes 1,600+ Models in Production with 2Tn+ Tokens

Recent developments in the evaluation of Large Language Models (LLMs) highlight the introduction of the 'BIGGEN BENCH' benchmark, which assesses 103 models across nine capabilities, including reasoning and multilingual skills. This benchmark aims to align AI assessments with human judgments through reliable and unbiased scoring methods utilizing synthetic data. In parallel, PortkeyAI has conducted a year-long analysis of LLM performance in production settings, examining over 1,600 models across more than 90 regions and involving over 2 trillion tokens. This research underscores the importance of understanding how LLMs function in real-world applications, with over 650 organizations relying on PortkeyAI's findings. Additionally, a new benchmark named 'AGORA BENCH' has been introduced by researchers from Carnegie Mellon University, KAIST, and the University of Washington, aimed at systematically evaluating language models as synthetic data generators.

#Large Language Models #PortkeyAI #Carnegie Mellon University #KAIST #University of Washington

Written with ChatGPT (GPT-4o mini).

Sources

Additional media

Image #1 for story new-biggen-bench-evaluates-103-language-models-across-9-capabilities-portkeyai-8b74d668

Image #2 for story new-biggen-bench-evaluates-103-language-models-across-9-capabilities-portkeyai-8b74d668