May 31, 07:28 PM

GPT-4 and Flan-PaLM Surpass Humans in 6th Order Theory of Mind Tasks

Recent research indicates that GPT-4 and Flan-PaLM, large language models (LLMs), have achieved adult-level and near adult-level performance on Theory of Mind (ToM) tasks. Notably, GPT-4 exceeds adult human performance on 6th order inferences. The study, published on May 29th on arXiv, involved 1,440 data points, though some data appeared noisy due to the small number of questions. The findings highlight the potential of LLMs in complex cognitive tasks, although the reliability of benchmarks for such tasks remains a topic of debate.

#Theory of Mind #arXiv

Written with ChatGPT (GPT-4o).

Sources

Ethan Mollick@emollick
2 years ago
LLMs “intelligence” is hard to benchmark, as we don’t have good benchmarks for human performance at complex tasks. Take theory-of-mind: several tests found GPT-4 beats humans, but another one finds a huge gap. Is it the testing structure? Prompting? Which is right? Hard to know. https://t.co/z9L3stRCDP
Hyunwoo Kim@hyunw__kim
2 years ago
⁉️ Let's check how GPT-4o, Gemini, Llama3, Mixtral, and Claude perform on theory of mind, shall we?🌟We report new results on Benchmark FANToM👻 - GPT-4o tops the chart by finally achieving score of 2.0/100 (vs. Human 87.5) - Huge boost for Gemini-1.5-flash compared to… https://t.co/rtuNsfEyIF https://t.co/GZWidSdIYs
Burny — Effective Omni@burny_tech
2 years ago
LLMs achieve adult human performance on higher-order theory of mind tasks GPT-4 and Flan-PaLM reach adult-level and near adult-level performance on ToM tasks overall, and that GPT-4 exceeds adult performance on 6th order inferences https://t.co/SsKbu4PbCo

Additional media

Image #1 for story gpt-4-flan-palm-surpass-humans-6th-order-theory-mind-tasks