Jul 19, 12:22 PM

Shanghai AI Laboratory and Tsinghua University Introduce NeedleBench for LLM Evaluation Across 1 Million Tokens

Researchers from Shanghai AI Laboratory and Tsinghua University have introduced NeedleBench, a new evaluation framework designed to test the long-context capabilities of large language models (LLMs). NeedleBench assesses bilingual long-context tasks across various intervals, ranging from 4,000 to over 1 million tokens. The framework includes a series of progressively more challenging tasks that span multiple length intervals, aiming to evaluate retrieval and reasoning capabilities. InternLM2.5-7B-Chat-1M has demonstrated impressive performance on NeedleBench 1000K, showcasing strong long-context retrieval and reasoning abilities. The announcement was made between July 17 and July 19.

#Shanghai AI Laboratory #Tsinghua University #NeedleBench

Written with ChatGPT (GPT-4o).