Safe Superintelligence News

J. Something 'just works' on the order of eg: train a predictive/imitative/generative AI on a human-generated dataset, and RLHF her to be unfailingly nice, generous to weaker entities, and determined to make the cosmos a lovely place.

K. Somebody discovers a new AI paradigm that's powerful enough and matures fast enough to beat deep learning to the punch, and the new paradigm is much much more alignable than giant inscrutable matrices of floating-point numbers.

C. Solving prosaic alignment on the first critical try is not as difficult, nor as dangerous, nor taking as much extra time, as Yudkowsky predicts; whatever effort is put forth by the leading coalition works inside of their lead time.

M. "We'll make the AI do our AI alignment homework" just works as a plan. (Eg the helping AI doesn't need to be smart enough to be deadly; the alignment proposals that most impress human judges are honest and truthful and successful.)

Something wonderful happens that isn't well-described by any option listed. (The semantics of this option may change if other options are added.)

A. Humanity successfully coordinates worldwide to prevent the creation of powerful AGIs for long enough to develop human intelligence augmentation, uploading, or some other pathway into transcending humanity's window of fragility.

G. It's impossible/improbable for something sufficiently smarter and more capable than modern humanity to be created, that it can just do whatever without needing humans to cooperate; nor does it successfully cheat/trick us.

E. Whatever strange motivations end up inside an unalignable AGI, or the internal slice through that AGI which codes its successor, they max out at a universe full of cheerful qualia-bearing life and an okay outcome for existing humans.

B. Humanity puts forth a tremendous effort, and delays AI for long enough, and puts enough desperate work into alignment, that alignment gets solved first.

I. The tech path to AGI superintelligence is naturally slow enough and gradual enough, that world-destroyingly-critical alignment problems never appear faster than previous discoveries generalize to allow safe further experimentation.

O. Early applications of AI/AGI drastically increase human civilization's sanity and coordination ability; enabling humanity to solve alignment, or slow down further descent into AGI, etc. (Not in principle mutex with all other answers.)

D. Early powerful AGIs realize that they wouldn't be able to align their own future selves/successors if their intelligence got raised further, and work honestly with humans on solving the problem in a way acceptable to both factions.

H. Many competing AGIs form an equilibrium whereby no faction is allowed to get too powerful, and humanity is part of this equilibrium and survives and gets a big chunk of cosmic pie.

L. Earth's present civilization crashes before powerful AGI, and the next civilization that rises is wiser and better at ops. (Exception to 'okay' as defined originally, will be said to count as 'okay' even if many current humans die.)

F. Somebody pulls off a hat trick involving blah blah acausal blah blah simulations blah blah, or other amazingly clever idea, which leads an AGI to put the reachable galaxies to good use despite that AGI not being otherwise alignable.

N. A crash project at augmenting human intelligence via neurotech, training mentats via neurofeedback, etc, produces people who can solve alignment before it's too late, despite Earth civ not slowing AI down much.

If you write an argument that breaks down the 'okay outcomes' into lots of distinct categories, without breaking down internal conjuncts and so on, Reality is very impressed with how disjunctive this sounds and allocates more probability.

OpenAI

xAI

Alphabet / Google

OpenAI (after January)

OpenAI (after February)

Google (after February)

xAI (After initial Grok 3 entry)

Google (after 2.5 release)

OpenAI (May or later)

Moonshot

DeepSeek

Anthropic

xAI (April or later)

Safe Superintelligence (SSI)

Meta

Microsoft (?)

Apple (?)

Mistral

Google Deepmind

OpenAI

Anthropic

XAI

Meta

Deepseek

Safe Superintelligence Inc

Manus AI

Huggingfaces

Tesla

Keen

CommaAI

stability.ai

Hugging Face

🤗 (hugging face)

HuggingFace (🤗)

~~Google~~ [SEE GOOGLE DEEPMIND]

Replit

Conjecture

Character.AI

Adept

NVIDIA

Apple

Midjourney

MosaicML

inflection.ai

Mistral AI

Moonshot AI

Huawei

Microsoft

ByteDance

ModelBest面壁智能

Singularity Net

High-Flyer

Alibaba

Reflection AI

Thinking Machines

Krantz

The company will be valued at >= $1 Billion according to a reputable news source (e.g. Forbes, Reuters, NYT)

The company will be valued at >= $10 Billion according to a reputable news source (e.g. Forbes, Reuters, NYT)

At least one of the founders (Ilya Sutskever, Daniel Gross, Daniel Levy) will leave the company

Zvi will mention the company in a blog post

Zvi will mention the company in a blog post in 2025

The company will be valued at >= $100 Million according to a reputable news source (e.g. Forbes, Reuters, NYT)

The company will raise more than $1 billion of capital

Ilya will remain at the company continuously until EOY 2025, or until the company is acquired/ceases to exist

The official SSI X account will have more than 100k followers

The majority of their compute will come from Nvidia GPUs

I will believe the company should have invested more in AI Safety relative to Capabilities at EOY 2025

Ilya will discuss the company on a podcast

The company will announce that their path to superintelligence involves self-play/synthetic data

The company will publish an assessment of the model’s dangerous capabilities (e.g. https://www.anthropic.com/news/frontier-threats-red-teaming-for-ai-safety)

The company will finish a training run reported to use more than 10^24 FLOP (e.g. by Epoch AI)

A majority of people believe that the company has been net-positive for the world according to a poll released at EOY 2025

Ilya will give a presentation on research done at the company

The company will include at least one image on its website

The company will announce that their model scores >= 85 MMLU

The company will announce that their model scores >= 50 GPQA

The company will invite independent researchers/orgs to do evals on their models

The company will finish a training run reported to use more than 10^25 FLOP (e.g. by Epoch AI)

The company will have at least 100 employees

The company will announce that their path to superintelligence involves creating an automated AI researcher

The company will announce research or models related to automated theorem proving (e.g. https://openai.com/index/generative-language-modeling-for-automated-theorem-proving/)

The company will be on track to build ASI by 2030, according to a Manifold poll conducted at EOY 2025

I will believe at EOY 2025 that the company has significantly advanced AI capabilities

The company will release a publicly available API for an AI model

The company will publish a Responsible Scaling Policy or similar document (e.g. OpenAI’s Preparedness Framework)

The company will publish research related specifically to Sparse Autoencoders

The official SSI X account will have more than 200k followers

I will meet an employee of the company in person (currently true for OAI, Anthropic, xAI but not Deepmind)

The company will sell any products or services before EOY 2025

The company will release a new AI or AI safety benchmark (e.g. MMLU, GPQA)

The company will announce that they are on track to develop superintelligence by EOY 2030 or earlier

The company will publish research which involves collaboration with at least 5 members of another leading AI lab (e.g. OAI, GDM, Anthropic, xAI)

The company will have a group of more than 10 people working on Mechanistic Interpretability

The company will release a chatbot or any other AI system which accepts text input

The company will release a model scoring >= 1300 elo in the chatbot arena leaderboard

The company will finish a training run reported to use more than 10^26 FLOP (e.g. by Epoch AI)

The company will open offices outside of the US and Israel

I will believe at EOY 2025 that the company has made significant progress in AI Alignment

I’ll work there (@mr_mino)

The company will announce a commitment to spend at least 20% of their compute on AI Safety/Alignment

The company will be listed as a “Frontier Lab” on https://ailabwatch.org/companies/

The company will be involved in a lawsuit

It will be reported that Nvidia is an investor in the company

The company’s model weights will be leaked/stolen

I will believe at EOY 2025 that the company has built an fully automated AI researcher

The company will make a GAN

The company will announce that their path to superintelligence involves continuous chain of thought

It’s reported that the company’s model scores >= 90 on the ARC-AGI challenge (public or private version)

The company will open source its model weights or training algorithms

It will be reported that a model produced by the company will self-exfiltrate, or attempt to do so

The official SSI X account will have more than 1M followers

The company will be valued at >= $100 Billion according to a reputable news source (e.g. Forbes, Reuters, NYT)

The phrase “Feel the AGI” or “Feel the ASI” will be published somewhere on the company website

The company will be reported to purchase at least $1 Billion in AI hardware, including cloud resources

Leopold Aschenbrenner will join the company

The company will advocate for a AI scaling pause or will endorse such a proposal (e.g. https://futureoflife.org/open-letter/pause-giant-ai-experiments/)

The company will have a public contract with the US government to develop some technology

The company will publish research related to Singular Learning Theory

Major algorithmic secrets (e.g architecture, training methods) will be leaked/stolen

The company will publish research related to Neural Turing Machines

The company’s AI will be involved in an accident which causes at least $10 million in damages

The company will release a model scoring in the top 3 of the chatbot arena leaderboard

The company will publish a research paper written entirely by their AI system

The company release a video generation demo made by their AI system

I will believe at EOY 2025 the company has made significant advances in robotics or manufacturing

Their model will be able to play Chess, Shogi, or Go at least as well as the best human players

There will be a public protest or boycott directed against the company with more than 100 members

The company will be closer to building ASI than any other AI Lab at EOY 2025, as judged by a manifold poll

The company’s model will independently solve an open mathematical conjecture created before 2024

The company will publish a peer-reviewed paper with more than 1000 citations

The company will be acquired by another company

Elon musk will be an investor of the company

The company will release a model that reaches the #1 rank in the Chatbot Arena (including sharing the #1 rank with other models when their confidence intervals overlap)

The company will release an app available on iPhone or android

The company will change its name

The company will be merged with or acquired by another company

The company will announce that they have created Superintelligence

The company will finish a training run reported to use more than 10^28 FLOP (e.g. by Epoch AI)

It will be reported that Sam Altman is an investor in the company

The company will build their own AI chips

Their model will be the first to get a gold medal or equivalent in IMO (International Mathematics Olympiad)

The company will finish a training run reported to use more than 10^29 FLOP (e.g. by Epoch AI)

The company will be reported to build a data center with a peak power consumption of >= 1 GW

The company will publish at least 5 papers in peer reviewed journals

The company will declare bankruptcy

The company will be reported to acquire an Aluminum manufacturing plant for its long term power contract

The company will be publicly traded

The company will finish a training run reported to use more than 10^27 FLOP (e.g. by Epoch AI)

The company will finish a training run reported to use more than 10^30 FLOP (e.g. by Epoch AI)

I'll work there (@AndrewG)

The company will be reported to build a data center with a peak power consumption of >=10 GW

The company will be reported to build a data center with a peak power consumption of >=100 GW

The company will be valued at >= $1 Trillion according to a reputable news source (e.g. Forbes, Reuters, NYT)

It will be for profit

It will reach a billion dollar valuation by 2025

At the start of 2026 the majority of their compute will be Nvidia GPUs

They receive Israeli gov investment

They will publicly demonstrate their AI or publish a paper by the start of 2026

At least 5 most recently OpenAI employees will join SSI inc by Jan 1st 2025

Will be purchased by or merge with another company by the end of 2026

Will launch a commercial product by the start of 2026

The Tel Aviv office (or any office in Israel) will no longer be important by EOY 2025

It will invent super intelligence (judged by me)

I'll work there

It will solve alignment

Daniel Gross (CEO, Safe Superintelligence)

Daniel Vincent Kramer

Koray Kavukcuoglu (DeepMind AI architect)

Soham Parekh (rumored Meta recruitment target)

John Schulman (OpenAI co-founder)

#149 – Tim LeBon on how altruistic perfectionism is self-defeating

#151 – Ajeya Cotra on accidentally teaching AI models to deceive us

#169 – Paul Niehaus on whether cash transfers cause economic growth, and keeping theft to acceptable levels

#170 – Santosh Harish on how air pollution is responsible for ~12% of global deaths — and how to get that number down

#150 – Tom Davidson on how quickly AI could transform the world

#152 – Joe Carlsmith on navigating serious philosophical confusion

#153 – Elie Hassenfeld on two big picture critiques of GiveWell's approach, and six lessons from their recent work

#154 – Rohin Shah on DeepMind and trying to fairly hear out both AI doomers and doubters

#155 – Lennart Heim on the compute governance era and what has to come after

#156 – Markus Anderljung on how to regulate cutting-edge AI models

#157 – Ezra Klein on existential risk from AI and what DC could do about it

#158 – Holden Karnofsky on how AIs might take over even if they're no smarter than humans, and his 4-part playbook for AI risk

#159 – Jan Leike on OpenAI's massive push to make superintelligence safe in 4 years or less

#161 – Michael Webb on whether AI will soon cause job loss, lower incomes, and higher inequality — or the opposite

#162 – Mustafa Suleyman on getting Washington and Silicon Valley to tame AI

#163 – Toby Ord on the perils of maximising the good that you do

#166 – Tantum Collins on what he's learned as an AI policy insider at the White House, DeepMind and elsewhere

#168 – Ian Morris on whether deep history says we're heading for an intelligence explosion

#171 – Alison Young on how top labs have jeopardised public health with repeated biosafety failures

#173 – Jeff Sebo on digital minds, and how to avoid sleepwalking into a major moral catastrophe

#174 – Nita Farahany on the neurotechnology already being used to convict criminals and manipulate workers

#176 – Nathan Labenz on the final push for AGI, understanding OpenAI's leadership drama, and red-teaming frontier models

#160 – Hannah Ritchie on why it makes sense to be optimistic about the environment

#164 – Kevin Esvelt on cults that want to kill everyone, stealth vs wildfire pandemics, and how he felt inventing gene drives

#165 – Anders Sandberg on war in space, whether civilisations age, and the best things possible in our universe

#175 – Lucia Coulter on preventing lead poisoning for $1.66 per child

#145 – Christopher Brown on why slavery abolition wasn't inevitable

#146 – Robert Long on why large language models like GPT (probably) aren't conscious

#167 – Seren Kell on the research gaps holding back alternative proteins from mass adoption

#172 – Bryan Caplan on why you should stop reading the news

#144 – Athena Aktipis on why cancer is actually one of the fundamental phenomena in our universe

#147 – Spencer Greenberg on stopping valueless papers from getting into top journals

K. Somebody discovers a new AI paradigm that's powerful enough and matures fast enough to beat deep learning to the punch, and the new paradigm is much much more alignable than giant inscrutable matrices of floating-point numbers.

I. The tech path to AGI superintelligence is naturally slow enough and gradual enough, that world-destroyingly-critical alignment problems never appear faster than previous discoveries generalize to allow safe further experimentation.

C. Solving prosaic alignment on the first critical try is not as difficult, nor as dangerous, nor taking as much extra time, as Yudkowsky predicts; whatever effort is put forth by the leading coalition works inside of their lead time.

B. Humanity puts forth a tremendous effort, and delays AI for long enough, and puts enough desperate work into alignment, that alignment gets solved first.

Something wonderful happens that isn't well-described by any option listed. (The semantics of this option may change if other options are added.)

M. "We'll make the AI do our AI alignment homework" just works as a plan. (Eg the helping AI doesn't need to be smart enough to be deadly; the alignment proposals that most impress human judges are honest and truthful and successful.)

A. Humanity successfully coordinates worldwide to prevent the creation of powerful AGIs for long enough to develop human intelligence augmentation, uploading, or some other pathway into transcending humanity's window of fragility.

E. Whatever strange motivations end up inside an unalignable AGI, or the internal slice through that AGI which codes its successor, they max out at a universe full of cheerful qualia-bearing life and an okay outcome for existing humans.

J. Something 'just works' on the order of eg: train a predictive/imitative/generative AI on a human-generated dataset, and RLHF her to be unfailingly nice, generous to weaker entities, and determined to make the cosmos a lovely place.

O. Early applications of AI/AGI drastically increase human civilization's sanity and coordination ability; enabling humanity to solve alignment, or slow down further descent into AGI, etc. (Not in principle mutex with all other answers.)

D. Early powerful AGIs realize that they wouldn't be able to align their own future selves/successors if their intelligence got raised further, and work honestly with humans on solving the problem in a way acceptable to both factions.

F. Somebody pulls off a hat trick involving blah blah acausal blah blah simulations blah blah, or other amazingly clever idea, which leads an AGI to put the reachable galaxies to good use despite that AGI not being otherwise alignable.

L. Earth's present civilization crashes before powerful AGI, and the next civilization that rises is wiser and better at ops. (Exception to 'okay' as defined originally, will be said to count as 'okay' even if many current humans die.)

G. It's impossible/improbable for something sufficiently smarter and more capable than modern humanity to be created, that it can just do whatever without needing humans to cooperate; nor does it successfully cheat/trick us.

H. Many competing AGIs form an equilibrium whereby no faction is allowed to get too powerful, and humanity is part of this equilibrium and survives and gets a big chunk of cosmic pie.

N. A crash project at augmenting human intelligence via neurotech, training mentats via neurofeedback, etc, produces people who can solve alignment before it's too late, despite Earth civ not slowing AI down much.

You are fooled by at least one option on this list, which out of many tries, ends up sufficiently well-aimed at your personal ideals / prejudices / the parts you understand less well / your own personal indulgences in wishful thinking.

YES

2027

2030 or later

2026

2028

2029

NO

YES