Aug 10, 10:44 AM

CamelAIOrg Releases Open-Source CRAB Framework for AI Benchmarking

Researchers from KAUST, UTokyo, CMU, Stanford, Harvard, Tsinghua, SUSTech, and Oxford, in collaboration with CamelAIOrg, have developed the CRAB framework, an AI framework designed for building LLM agent benchmark environments in a Python-centric way. CRAB, which stands for Cross-environment Agent Benchmark, aims to become a general-purpose agent benchmark framework for Multimodal Language Model (MLM) agents. The framework includes CRAB Benchmark-v0, developed using the CRAB framework, which features 100 tasks across two environments, Ubuntu and Android. It provides an end-to-end and easy-to-use framework to build multimodal agents, operate environments, and create benchmarks to evaluate them. The CRAB framework is now open-sourced, allowing agents to control devices such as mobile phones, laptops, or desktops from a single prompt.

#KAUST #UTokyo #CMU #Stanford #Harvard #Tsinghua #SUSTech #Oxford #CamelAIOrg #Multimodal Language Model #CRAB #Ubuntu #Android

Written with ChatGPT (GPT-4o).

CamelAIOrg Releases Open-Source CRAB Framework for AI Benchmarking

Sources

Additional media

Similar Stories