Benchmark News

Hut 8 Unveils 1.5-GW U.S. Build-Out; Benchmark Ups Target

3 months ago

Semtech Gains as Q2 Beat Spurs Cluster of Price-Target Hikes

Company Earnings

Stocks

Business

Semtech Gains as Q2 Beat Spurs Cluster of Price-Target Hikes

3 months ago

Madrona Unveils 2025 IA40, Spotlighting 40 Rising AI Startups

AI Products

AI Fundraising

Madrona Unveils 2025 IA40, Spotlighting 40 Rising AI Startups

3 months ago

Will an AI achieve >85% performance on the FrontierMath benchmark before 2028?

Nov 8, 8:25 PMJan 2, 7:59 AM

54.79%chance

255259969

OptionVotes

YES

5157

4769

Which AI companies will release a top-scoring LLM on the Scale AI Coding benchmark in 2025?

Dec 28, 7:34 AMJan 1, 7:59 AM

2779803

OptionProbability

OpenAI

Google

Anthropic

xAI

By 2029, will any AI be able to read a novel and reliably answer questions about it? (Gary Marcus benchmark #2)

Sep 16, 8:34 PMJan 1, 8:00 AM

89.93%chance

11745514

OptionVotes

YES

2062

1522

Will an AI achieve >85% performance on the FrontierMath benchmark before 2027?

Nov 8, 11:52 PMJan 2, 7:59 AM

38.45%chance

8537707

OptionVotes

YES

1265

790

By 2029 will any AI be able to watch a movie and accurately tell you what is going on? (Gary Marcus benchmark #1)

Sep 16, 8:33 PMJan 1, 8:00 AM

86.92%chance

9025906

OptionVotes

YES

1895

1106

In 2029, will any AI be able to construct "reasonably" bug-free code of >= 10k LOC from a natural language specification? (Gary Marcus benchmark #4)

Sep 16, 8:43 PMJan 1, 8:00 AM

81.61%chance

11225232

OptionVotes

YES

1668

729

In 2029, will any AI be able to take an arbitrary proof in the mathematical literature and translate it into a form suitable for symbolic verification? (Gary Marcus benchmark #5)

Sep 16, 8:47 PMJan 1, 8:00 AM

82.27%chance

10116414

OptionVotes

YES

2263

656

In 2029, will any AI be able to work as a competent cook in an arbitrary kitchen? (Gary Marcus benchmark #3)

Sep 16, 8:37 PMJan 1, 8:00 AM

59.79%chance

11415066

OptionVotes

YES

2017

1002

On January 1, 2027, a Transformer-like model will continue to hold the state-of-the-art position in most benchmark

Jun 13, 9:21 AMDec 31, 9:59 PM

84.23%chance

415711

OptionVotes

YES

2299

317

Will an AI achieve >80% performance on the FrontierMath benchmark before 2027?

Feb 7, 7:34 PMJan 1, 7:59 AM

40.68%chance

425677

OptionVotes

YES

1208

828

Will at least 3 of the Gary Marcus benchmark questions resolve YES?

Sep 16, 9:19 PMJan 1, 8:00 AM

89.48%chance

413065

OptionVotes

YES

1830

824

Benchmark Gap #3: Once a model achieves superhuman performance on a competitive programming benchmark, will it be less than 2 years before there are "entry level" AI programmers in industry use?

Apr 8, 6:06 PMJan 1, 4:59 AM

73.23%chance

14560

OptionVotes

YES

1156

931

Latest stories

Aug 28, 2025
Aug 28, 2025
VC
AI Products
AI Fundraising
Madrona Unveils 2025 IA40, Spotlighting 40 Rising AI Startups
Authors
8
3 months ago
Aug 27, 2025
Aug 27, 2025
Stocks
Mining
Business
Hut 8 Unveils 1.5-GW U.S. Build-Out; Benchmark Ups Target
Authors
5
3 months ago
Aug 26, 2025
Aug 26, 2025
Company Earnings
Stocks
Business
Semtech Gains as Q2 Beat Spurs Cluster of Price-Target Hikes
Authors
9
3 months ago

Join our channels

Get the latest stories live on any device.

WhatsApp
Top Stories
Join
Telegram
Top Stories
Join

Benchmark News

Top stories

Prediction markets for Benchmark

Prediction markets for Benchmark

Prediction markets for Benchmark

Will an AI achieve >85% performance on the FrontierMath benchmark before 2028?

Which AI companies will release a top-scoring LLM on the Scale AI Coding benchmark in 2025?

By 2029, will any AI be able to read a novel and reliably answer questions about it? (Gary Marcus benchmark #2)

Will an AI achieve >85% performance on the FrontierMath benchmark before 2027?

By 2029 will any AI be able to watch a movie and accurately tell you what is going on? (Gary Marcus benchmark #1)

In 2029, will any AI be able to construct "reasonably" bug-free code of >= 10k LOC from a natural language specification? (Gary Marcus benchmark #4)

In 2029, will any AI be able to take an arbitrary proof in the mathematical literature and translate it into a form suitable for symbolic verification? (Gary Marcus benchmark #5)

In 2029, will any AI be able to work as a competent cook in an arbitrary kitchen? (Gary Marcus benchmark #3)

On January 1, 2027, a Transformer-like model will continue to hold the state-of-the-art position in most benchmark

Will an AI achieve >80% performance on the FrontierMath benchmark before 2027?

Will at least 3 of the Gary Marcus benchmark questions resolve YES?

Benchmark Gap #3: Once a model achieves superhuman performance on a competitive programming benchmark, will it be less than 2 years before there are "entry level" AI programmers in industry use?

Latest stories

Join our channels

WhatsApp

Telegram