Benchmark News

Hut 8 Unveils 1.5-GW U.S. Build-Out; Benchmark Ups Target

1 year ago

Semtech Gains as Q2 Beat Spurs Cluster of Price-Target Hikes

Company Earnings

Stocks

Business

Semtech Gains as Q2 Beat Spurs Cluster of Price-Target Hikes

1 year ago

Madrona Unveils 2025 IA40, Spotlighting 40 Rising AI Startups

AI Products

AI Fundraising

Madrona Unveils 2025 IA40, Spotlighting 40 Rising AI Startups

1 year ago

By 2029, will any AI be able to read a novel and reliably answer questions about it? (Gary Marcus benchmark #2)

Sep 16, 8:34 PMJan 1, 8:00 AM

88.92%chance

12755508

OptionVotes

YES

1879

1543

By 2029 will any AI be able to watch a movie and accurately tell you what is going on? (Gary Marcus benchmark #1)

Sep 16, 8:33 PMJan 1, 8:00 AM

93.73%chance

9630812

OptionVotes

YES

3607

937

In 2029, will any AI be able to construct "reasonably" bug-free code of >= 10k LOC from a natural language specification? (Gary Marcus benchmark #4)

Sep 16, 8:43 PMJan 1, 8:00 AM

92.21%chance

11626924

OptionVotes

YES

3190

523

In 2029, will any AI be able to take an arbitrary proof in the mathematical literature and translate it into a form suitable for symbolic verification? (Gary Marcus benchmark #5)

Sep 16, 8:47 PMJan 1, 8:00 AM

78.47%chance

10719653

OptionVotes

YES

1971

727

In 2029, will any AI be able to work as a competent cook in an arbitrary kitchen? (Gary Marcus benchmark #3)

Sep 16, 8:37 PMJan 1, 8:00 AM

56.49%chance

11915639

OptionVotes

YES

1904

1083

On January 1, 2027, a Transformer-like model will continue to hold the state-of-the-art position in most benchmark

Jun 13, 9:21 AMDec 31, 9:59 PM

94.45%chance

477995

OptionVotes

YES

3759

163

Will at least 3 of the Gary Marcus benchmark questions resolve YES?

Sep 16, 9:19 PMJan 1, 8:00 AM

93%chance

444365

OptionVotes

YES

2605

751

Will a frontier model score above 90% on the APEX-SWE benchmark before 2028?

Mar 26, 10:45 PMJan 2, 7:59 AM

71.03%chance

91028

OptionVotes

YES

1566

639

Benchmark Gap #3: Once a model achieves superhuman performance on a competitive programming benchmark, will it be less than 2 years before there are "entry level" AI programmers in industry use?

Apr 8, 6:06 PMJan 1, 4:59 AM

78%chance

16943

OptionVotes

YES

1382

859

Benchmark Gap #1: Once we have a language model that achieves expert human performance on all current major NLP benchmarks, how many years will it be before we have an AI with human-level language skills?

Oct 27, 5:11 PMJan 1, 8:00 AM

8.61%chance

11715

OptionProbability

YES

1249

958

Will "Every Benchmark is Broken" make the top fifty posts in LessWrong's 2026 Annual Review?

Jul 4, 11:06 PMFeb 1, 12:00 AM

14.52%chance

5182

OptionVotes

YES

1006

964

What will the top score on Epoch AI's Chess Puzzles benchmark be on November 30, 2026?

Nov 1, 4:26 AMNov 30, 11:59 PM

OptionProbability

Below 78%

78% to below 82%

82% to below 88%

88% or higher

Latest stories

Aug 28, 2025
Aug 28, 2025
VC
AI Products
AI Fundraising
Madrona Unveils 2025 IA40, Spotlighting 40 Rising AI Startups
Authors
8
1 year ago
Aug 27, 2025
Aug 27, 2025
Stocks
Mining
Business
Hut 8 Unveils 1.5-GW U.S. Build-Out; Benchmark Ups Target
Authors
5
1 year ago
Aug 26, 2025
Aug 26, 2025
Company Earnings
Stocks
Business
Semtech Gains as Q2 Beat Spurs Cluster of Price-Target Hikes
Authors
9
1 year ago

Join our channels

Get the latest stories live on any device.

WhatsApp
Top Stories
Join
Telegram
Top Stories
Join

Benchmark News

Top stories

Prediction markets for Benchmark

Prediction markets for Benchmark

Prediction markets for Benchmark

By 2029, will any AI be able to read a novel and reliably answer questions about it? (Gary Marcus benchmark #2)

By 2029 will any AI be able to watch a movie and accurately tell you what is going on? (Gary Marcus benchmark #1)

In 2029, will any AI be able to construct "reasonably" bug-free code of >= 10k LOC from a natural language specification? (Gary Marcus benchmark #4)

In 2029, will any AI be able to take an arbitrary proof in the mathematical literature and translate it into a form suitable for symbolic verification? (Gary Marcus benchmark #5)

In 2029, will any AI be able to work as a competent cook in an arbitrary kitchen? (Gary Marcus benchmark #3)

On January 1, 2027, a Transformer-like model will continue to hold the state-of-the-art position in most benchmark

Will at least 3 of the Gary Marcus benchmark questions resolve YES?

Will a frontier model score above 90% on the APEX-SWE benchmark before 2028?

Benchmark Gap #3: Once a model achieves superhuman performance on a competitive programming benchmark, will it be less than 2 years before there are "entry level" AI programmers in industry use?

Benchmark Gap #1: Once we have a language model that achieves expert human performance on all *current* major NLP benchmarks, how many years will it be before we have an AI with human-level language skills?

Will "Every Benchmark is Broken" make the top fifty posts in LessWrong's 2026 Annual Review?

What will the top score on Epoch AI's Chess Puzzles benchmark be on November 30, 2026?

Latest stories

Join our channels

WhatsApp

Telegram

Benchmark Gap #1: Once we have a language model that achieves expert human performance on all current major NLP benchmarks, how many years will it be before we have an AI with human-level language skills?