Future House Releases LAB-Bench with 2000 Evaluations; Claude 3.5 Sonnet Leads

Future House has released LAB-Bench, a comprehensive set of over 2000 evaluations designed to test the capabilities of language models and agents in performing scientific research tasks in biology. Public models have generally underperformed compared to PhD and postdoc-level humans on nearly all tasks. Claude 3.5 Sonnet has emerged as the leading model, although it still has significant room for improvement. LAB-Bench is the first evaluation set aimed at assessing whether models can conduct scientific research rather than merely knowing scientific trivia. This initiative is seen as a crucial step towards enhancing the role of language models as scientific collaborators. Additionally, SciCode, a benchmark challenging language models to code solutions for scientific problems from advanced papers, shows that GPT-4 and Sonnet 3.5 achieve less than 5% accuracy, with about 10% of its challenges based on Nobel-winning research.