Coding News

AI Modeling

Anthropic’s Claude Opus 4.1 Sweeps LM Arena AI Benchmarks

4 months ago

Which AI companies will release a top-scoring LLM on the Scale AI Coding benchmark in 2025?

Dec 28, 7:34 AMJan 1, 7:59 AM

2779803

OptionProbability

OpenAI

Google

Anthropic

xAI

AI coding agent forecasts from my blog post, through July 4, 2026

Jul 5, 7:26 AMJul 4, 11:59 PM

3473575

OptionProbability

Coding agents will close the loop

Recursively improving coding agents will succeed in the market

Coding agents will flip the initiative

The specific model will not matter as much as today; the network of agents will be important

The multi-agent paradigm will win

“Live learning” will be standard

xAI will gain a sizable lead in model quality

Will Meta have a "mid-level" AI engineer that can write code by the end of 2025?

Jan 13, 7:06 PMDec 31, 8:59 PM

9.34%chance

7655933

OptionVotes

YES

31227

3511

In 2029, will any AI be able to construct "reasonably" bug-free code of >= 10k LOC from a natural language specification? (Gary Marcus benchmark #4)

Sep 16, 8:43 PMJan 1, 8:00 AM

81.61%chance

11225232

OptionVotes

YES

1668

729

Will SotA on PaperBench (Code-Dev) surpass 75% in 2025?

Apr 7, 1:59 PMJan 15, 11:59 PM

14.31%chance

117255

OptionVotes

YES

12235

2473

EOY 2025: Will open LLMs perform at least as well as 50 Elo below closed-source LLMs on coding?

Jul 27, 10:58 PMJan 1, 4:59 AM

72.08%chance

246791

OptionVotes

YES

1607

622

By 2030, will over 50% of software development projects be primarily created by AI, with minimal human coding?

Jan 20, 10:38 PMJan 2, 7:59 AM

69.62%chance

344443

OptionVotes

YES

1628

301

Will the US Government issue a statement criticising the Code of Practice of the EU AI Act in 2025?

May 16, 10:35 PMDec 31, 11:21 PM

67.04%chance

303185

OptionVotes

YES

2582

1615

GPT-4 #1: Conditional on being able to use a REPL, will GPT-4 be able to be prompted to write naively malicious code?

Nov 15, 6:21 PMJan 1, 8:00 AM

53.66%chance

321555

OptionVotes

YES

1225

592

Will AI be able to write, compile, and unit test a single .c file to reproduce GPT-2 training from PyTorch code by 2026?

Apr 9, 4:13 PMApr 9, 6:29 PM

70.13%chance

291047

OptionVotes

YES

1107

930

Will the "Stepped Up Basis" rule be eliminated in the US federal tax code before 2035?

Aug 28, 8:27 PMSep 5, 3:59 AM

31.16%chance

2500

OptionVotes

YES

1486

673

Conditional on AI taking coding jobs, will devs brought in to clean up after an AI complain the AI code is unreadable?

Apr 8, 11:04 AMApr 8, 10:59 PM

74.74%chance

18329

OptionVotes

YES

1081

960

Latest stories

Aug 18, 2025
Aug 18, 2025
AI Modeling
AI
Anthropic’s Claude Opus 4.1 Sweeps LM Arena AI Benchmarks
Authors
4
4 months ago

Join our channels

Get the latest stories live on any device.

WhatsApp
Top Stories
Join
Telegram
Top Stories
Join

Coding News

Top stories

Prediction markets for Coding

Prediction markets for Coding

Prediction markets for Coding

Which AI companies will release a top-scoring LLM on the Scale AI Coding benchmark in 2025?

AI coding agent forecasts from my blog post, through July 4, 2026

Will Meta have a "mid-level" AI engineer that can write code by the end of 2025?

In 2029, will any AI be able to construct "reasonably" bug-free code of >= 10k LOC from a natural language specification? (Gary Marcus benchmark #4)

Will SotA on PaperBench (Code-Dev) surpass 75% in 2025?

EOY 2025: Will open LLMs perform at least as well as 50 Elo below closed-source LLMs on coding?

By 2030, will over 50% of software development projects be primarily created by AI, with minimal human coding?

Will the US Government issue a statement criticising the Code of Practice of the EU AI Act in 2025?

GPT-4 #1: Conditional on being able to use a REPL, will GPT-4 be able to be prompted to write naively malicious code?

Will AI be able to write, compile, and unit test a single .c file to reproduce GPT-2 training from PyTorch code by 2026?

Will the "Stepped Up Basis" rule be eliminated in the US federal tax code before 2035?

Conditional on AI taking coding jobs, will devs brought in to clean up after an AI complain the AI code is unreadable?

Latest stories

Join our channels

WhatsApp

Telegram