Scale AI News

Humanity coordinates to prevent the creation of potentially-unsafe AIs.

We create a truth economy. https://manifold.markets/Krantz/is-establishing-a-truth-economy-tha?r=S3JhbnR6

Eliezer finally listens to Krantz.

Other

Yudkowsky is trying to solve the wrong problem using the wrong methods based on a wrong model of the world derived from poor thinking and fortunately all of his mistakes have failed to cancel out

Alignment is not properly solved, but core human values are simple enough that partial alignment techniques can impart these robustly. Despite caring about other things, it is relatively cheap for AGI to satisfy human values.

Someone solves agent foundations

Ethics turns out to be a precondition of superintelligence

Because of quantum immortality we will observe only the worlds where AI will not kill us (assuming that s-risks chances are even smaller, it is equal to ok outcome).

Orthogonality Thesis is false.

Sheer Dumb Luck. The aligned AI agrees that alignment is hard, any Everett branches in our neighborhood with slightly different AI models or different random seeds are mostly dead.

Alignment is impossible. Sufficiently smart AIs know this and thus won't improve themselves and won't create successor AIs, but will instead try to prevent existence of smarter AIs, just as smart humans do.

AI systems good at finding alignment solutions to capable systems (via some solution in the space of alignment solutions, supposing it is non-null, and that we don't have a clear trajectory to get to) have find some solution to alignment.

Humans become transhuman through other means before AGI happens

Alignment is unsolvable. AI that cares enough about its goal to destroy humanity is also forced to take it slow trying to align its future self, preventing run-away.

Aliens invade and stop bad |AI from appearing

Techniques along the lines outlined by Collin Burns turn out to be sufficient for alignment (AIs/AGIs are made truthful enough that they can be used to get us towards full alignment)

A smaller AI disaster causes widespread public panic about AI, making it a bad legal or PR move to invest in powerful AIs without also making nearly-crippling safety guarantees

There is a natural limit of effectiveness of intelligence, like diminishing returns, and it is on the level IQ=1000. AIs have to collaborate with humans.

AGI is never built (indefinite global moratorium)

Co-operative AI research leads to the training of agents with a form of pro-social concern that generalises to out of distribution agents with hidden utilities, i.e. humans.

AIs will not have utility functions (in the same sense that humans do not), their goals such as they are will be relatively humanlike, and they will be "computerish" and generally weakly motivated compared to humans.

Something to do with self-other overlap, which Eliezer called "Not obviously stupid" - https://www.lesswrong.com/posts/hzt9gHpNwA2oHtwKX/self-other-overlap-a-neglected-approach-to-ai-alignment?commentId=WapHz3gokGBd3KHKm

Pascals mugging: it’s not okay in 99.9% of the worlds but the 0.1% are so much better that the combined EV of AGI for the multiverse is positive

The Super-Strong Self Sampling Assumption (SSSSA) is true. If superintelligence is possible, "I" will become the superintelligence.

AI control gets us helpful enough systems without being deadly

an aligned AGI is built and the aligned AGI prevents the creation of any unaligned AGI.

I've been a good bing 😊

We make risk-conservative requests to extract alignment-related work out of AI-systems that were boxed prior to becoming superhuman. We somehow manage to achieve a positive feedback-loop in alignment/verification-abilities.

The response to AI advancements or failures makes some governments delay the timelines

Far more interesting problems to solve than take over the world and THEN solve them. The additional kill all humans step is either not a low-energy one or just by chance doesn't get converged upon.

AIs make "proof-like" argumentation for why output does/is what we want. We manage to obtain systems that *predict* human evaluations of proof-steps, and we manage to find/test/leverage regularities for when humans *aren't* fooled.

A lot of humans participate in a slow scalable oversight-style system, which is pivotally used/solves alignment enough

Something less inscrutable than matrices works fast enough

There’s some cap on the value extractible from the universe and we already got the 20%

SHA3-256: 1f90ecfdd02194d810656cced88229c898d6b6d53a7dd6dd1fad268874de54c8

Robot Love!!

AI thinks it is in a simulation controlled by Roko's basilisk

The human brain is the perfect arrangement of atoms for a "takeover the world" agent, so AGI has no advantage over us in that task.

Aligned AI is more economically valuable than unaligned AI. The size of this gap and the robustness of alignment techniques required to achieve it scale up with intelligence, so economics naturally encourages solving alignment.

Humans and human tech (like AI) never reach singularity, and whatever eats our lightcone instead (like aliens) happens to create an "okay" outcome

AIs never develop coherent goals

Rolf Nelson's idea that we make precommitment to simulate all possible bad AIs works – and keeps AI in check.

Nick Bostrom's idea (Hail Mary) that AI will preserve humans to trade with possible aliens works

For some reason, the optimal strategy for AGIs is just to head somewhere with far more resources than Earth, as fast as possible. All unaligned AGIs immediately leave, and, for some reason, do not leave anything behind that kills us.

An AI that is not fully superior to humans launches a failed takeover, and the resulting panic convinces the people of the world to unite to stop any future AI development.

We're inside of a simulation created by an entity that has values approximately equal to ours, and it intervenes and saves us from unaligned AI.

God exists and stops the AGI

Someone at least moderately sane leads a campaign, becomes in charge of a major nation, and starts a secret project with enough resources to solve alignment, because it turns out there's a way to convert resources into alignment progress.

Someone creates AGI(s) in a box, and offers to split the universe. They somehow find a way to arrange this so that the AGI(s) cannot manipulate them or pull any tricks, and the AGI(s) give them instructions for safe pivotal acts.

Someone understands how minds work enough to successfully build and use one directed at something world-savingly enough

Dolphins, or some other species, but probably dolphins, have actually been hiding in the shadows, more intelligent than us, this whole time. Their civilization has been competent enough to solve alignment long before we can create an AGI.

AGIs' takeover attempts are defeated by Michael Biehn with a pipe bomb.

Eliezer funds the development of controllable nanobots that melt computer circuitry, and they destroy all computers, preventing the Singularity. If Eliezer's past self from the 90s could see this, it would be so so so soooo hilarious.

Several AIs are created but they move in opposite directions with near light speed, so they never interacts. At least one of them is friendly and it gets a few percents of the total mass of the universe.

Unfriendly AIs choose to advance not outwards but inwards, and form a small blackhole which helps them to perform more calculations than could be done with the whole mass of the universe. For external observer such AIs just disappear.

Any sufficiently advance AI halts because it wireheads itself or halts for some other reasons. This puts a natural limit on AI's intelligence, and lower intelligence AIs are not that dangerous.

Social contagion causes widespread public panic about AI, making it a bad legal or PR move to invest in powerful AIs without also making nearly-crippling safety guarantees

Getting things done in Real World is as hard for AGI as it is for humans. AGI needs human help, but aligning humans is as impossible as aligning AIs. Humans and AIs create billions of competing AGIs with just as many goals.

Development and deployment of advanced AI occurs within a secure enclave which can only be interfaced with via a decentralized governance protocol

Friendly AI more likely to resurrect me than paperclipper or suffering maximiser. Because of quantum immortality I will find myself eventually resurrected. Friendly AIs will wage a multiverse wide war against s-risks, s-risks are unlikely.

High-level self-improvement (rewriting code) is intrinsically risky process, so AIs will prefer low level and slow self-improvement (learning), thus AIs collaborating with humans will have advantage. Ends with posthumans ecosystem.

Human consciousness is needed to collapse wave function, and AI can't do it. Thus humans should be preserved and they may require complete friendliness in exchange (or they will be unhappy and produce bad collapses)

Power dynamics stay multi-polar. Partly easy copying of SotA performance, bigger projects need high coordination, and moderate takeoff speed. And "military strike on all society" remains an abysmal strategy for practically all entities.

First AI is actually a human upload (maybe LLM-based model of person) AND it will be copies many times to form weak AI Nanny which prevents creation of other AIs.

Nanotech is difficult without experiments, so no mail order AI Grey Goo; Humans will be the main workhorse of AI everywhere. While they will be exploited, this will be like normal life from inside

ASI needs not your atoms but information. Humans will live very interesting lives.

Something else

Moral Realism is true, the AI discovers this and the One True Morality is human-compatible.

Valence realism is true. AGI hacks itself to experiencing every possible consciousness and picks the best one (for everyone)

AGI develops natural abstractions sufficiently similar to ours that it is aligned with us by default

AGI discovers new physics and exits to another dimension (like the creatures in Greg Egan’s Crystal Nights).

Alien Information Theory is true (this is discovered by experiments with sustained hours/days long DMT trips). The aliens have solved alignment and give us the answer.

AGI executes a suicide plan that destroys itself and other potential AGIs, but leaves humans in an okay outcome.

Multipolar AGI Agents run wild on the internet, hacking/breaking everything, causing untold economic damage but aren't focused enough to manipulate humans to achieve embodiment. In the aftermath, humanity becomes way saner about alignment.

Some form of objective morality is true, and any sufficiently intelligent agent automatically becomes benevolent.

"Corrigibility" is a bit more mathematically straightforward than was initially presumed, in the sense that we can expect it to occur, and is relatively easy to predict, even under less-than-ideal conditions.

Either the "strong form" of the Orthogonality Thesis is false, or "Goal-directed agents are as tractable as their goals" is true while goal-sets which are most threatening to humanity are relatively intractable.

A concerted effort targets an agent at a capability plateau which is adequate to defer the hard parts of the problem until later. The necessary near-term problems to solve didn't depend on deeply modeling human values.

Almost all human values are ex post facto rationalizations and enough humans survive to do what they always do

The company will be valued at >= $1 Billion according to a reputable news source (e.g. Forbes, Reuters, NYT)

The company will be valued at >= $10 Billion according to a reputable news source (e.g. Forbes, Reuters, NYT)

At least one of the founders (Ilya Sutskever, Daniel Gross, Daniel Levy) will leave the company

Zvi will mention the company in a blog post

Zvi will mention the company in a blog post in 2025

The company will be valued at >= $100 Million according to a reputable news source (e.g. Forbes, Reuters, NYT)

The company will raise more than $1 billion of capital

Ilya will remain at the company continuously until EOY 2025, or until the company is acquired/ceases to exist

The official SSI X account will have more than 100k followers

The majority of their compute will come from Nvidia GPUs

I will believe the company should have invested more in AI Safety relative to Capabilities at EOY 2025

Ilya will discuss the company on a podcast

The company will announce that their path to superintelligence involves self-play/synthetic data

The company will publish an assessment of the model’s dangerous capabilities (e.g. https://www.anthropic.com/news/frontier-threats-red-teaming-for-ai-safety)

The company will finish a training run reported to use more than 10^24 FLOP (e.g. by Epoch AI)

A majority of people believe that the company has been net-positive for the world according to a poll released at EOY 2025

Ilya will give a presentation on research done at the company

The company will include at least one image on its website

The company will announce that their model scores >= 85 MMLU

The company will announce that their model scores >= 50 GPQA

The company will invite independent researchers/orgs to do evals on their models

The company will finish a training run reported to use more than 10^25 FLOP (e.g. by Epoch AI)

The company will have at least 100 employees

The company will announce that their path to superintelligence involves creating an automated AI researcher

The company will announce research or models related to automated theorem proving (e.g. https://openai.com/index/generative-language-modeling-for-automated-theorem-proving/)

The company will be on track to build ASI by 2030, according to a Manifold poll conducted at EOY 2025

I will believe at EOY 2025 that the company has significantly advanced AI capabilities

The company will release a publicly available API for an AI model

The company will publish a Responsible Scaling Policy or similar document (e.g. OpenAI’s Preparedness Framework)

The company will publish research related specifically to Sparse Autoencoders

The official SSI X account will have more than 200k followers

I will meet an employee of the company in person (currently true for OAI, Anthropic, xAI but not Deepmind)

The company will sell any products or services before EOY 2025

The company will release a new AI or AI safety benchmark (e.g. MMLU, GPQA)

The company will announce that they are on track to develop superintelligence by EOY 2030 or earlier

The company will publish research which involves collaboration with at least 5 members of another leading AI lab (e.g. OAI, GDM, Anthropic, xAI)

The company will have a group of more than 10 people working on Mechanistic Interpretability

The company will release a chatbot or any other AI system which accepts text input

The company will release a model scoring >= 1300 elo in the chatbot arena leaderboard

The company will finish a training run reported to use more than 10^26 FLOP (e.g. by Epoch AI)

The company will open offices outside of the US and Israel

I will believe at EOY 2025 that the company has made significant progress in AI Alignment

I’ll work there (@mr_mino)

The company will announce a commitment to spend at least 20% of their compute on AI Safety/Alignment

The company will be listed as a “Frontier Lab” on https://ailabwatch.org/companies/

The company will be involved in a lawsuit

It will be reported that Nvidia is an investor in the company

The company’s model weights will be leaked/stolen

I will believe at EOY 2025 that the company has built an fully automated AI researcher

The company will make a GAN

The company will announce that their path to superintelligence involves continuous chain of thought

It’s reported that the company’s model scores >= 90 on the ARC-AGI challenge (public or private version)

The company will open source its model weights or training algorithms

It will be reported that a model produced by the company will self-exfiltrate, or attempt to do so

The official SSI X account will have more than 1M followers

The company will be valued at >= $100 Billion according to a reputable news source (e.g. Forbes, Reuters, NYT)

The phrase “Feel the AGI” or “Feel the ASI” will be published somewhere on the company website

The company will be reported to purchase at least $1 Billion in AI hardware, including cloud resources

Leopold Aschenbrenner will join the company

The company will advocate for a AI scaling pause or will endorse such a proposal (e.g. https://futureoflife.org/open-letter/pause-giant-ai-experiments/)

The company will have a public contract with the US government to develop some technology

The company will publish research related to Singular Learning Theory

Major algorithmic secrets (e.g architecture, training methods) will be leaked/stolen

The company will publish research related to Neural Turing Machines

The company’s AI will be involved in an accident which causes at least $10 million in damages

The company will release a model scoring in the top 3 of the chatbot arena leaderboard

The company will publish a research paper written entirely by their AI system

The company release a video generation demo made by their AI system

I will believe at EOY 2025 the company has made significant advances in robotics or manufacturing

Their model will be able to play Chess, Shogi, or Go at least as well as the best human players

There will be a public protest or boycott directed against the company with more than 100 members

The company will be closer to building ASI than any other AI Lab at EOY 2025, as judged by a manifold poll

The company’s model will independently solve an open mathematical conjecture created before 2024

The company will publish a peer-reviewed paper with more than 1000 citations

The company will be acquired by another company

Elon musk will be an investor of the company

The company will release a model that reaches the #1 rank in the Chatbot Arena (including sharing the #1 rank with other models when their confidence intervals overlap)

The company will release an app available on iPhone or android

The company will change its name

The company will be merged with or acquired by another company

The company will announce that they have created Superintelligence

The company will finish a training run reported to use more than 10^28 FLOP (e.g. by Epoch AI)

It will be reported that Sam Altman is an investor in the company

The company will build their own AI chips

Their model will be the first to get a gold medal or equivalent in IMO (International Mathematics Olympiad)

The company will finish a training run reported to use more than 10^29 FLOP (e.g. by Epoch AI)

The company will be reported to build a data center with a peak power consumption of >= 1 GW

The company will publish at least 5 papers in peer reviewed journals

The company will declare bankruptcy

The company will be reported to acquire an Aluminum manufacturing plant for its long term power contract

The company will be publicly traded

The company will finish a training run reported to use more than 10^27 FLOP (e.g. by Epoch AI)

The company will finish a training run reported to use more than 10^30 FLOP (e.g. by Epoch AI)

I'll work there (@AndrewG)

The company will be reported to build a data center with a peak power consumption of >=10 GW

The company will be reported to build a data center with a peak power consumption of >=100 GW

The company will be valued at >= $1 Trillion according to a reputable news source (e.g. Forbes, Reuters, NYT)

OpenAI

Google

Anthropic

DeepSeek

Any Chinese Company

Meta

We could map the entirety of analytic philosophy using this question.

The list of answers to this question form a constitution of truth that can be aligned decentrally by a free market.

A network state could use a predictive market constitution to define its smart social contracts.

Large scale AI systems should have intrinsic guardrail behaviors that no one actor can override.

This prediction is effectively the same as this one and nobody can explain why this mechanism isn't able to align AI at scale. https://manifold.markets/Krantz/krantz-mechanism-demonstration

Betting on philosophy seems like a fun way to (1) learn philosophy (2) contribute to a transhumanist utopia world where our net incomes are highly correlated with how much beneficial stuff we taught the public domain AI.

Constitutions play a critical role in the frontier methods for aligning AI.

A good AI should not kill people.

There would be a dramatic positive change in the world if teenagers and homeless folks could earn crypto on an app they download for free to argue philosophy with an AI until they can either prove the AI is right or prove it is wrong.

Philosophy is primarily the pursuit of defining language.

A duty to reason is the foundation for goodness.

I have free will.

A good AI should not infringe on the rights, autonomy or property of humans.

We should stop doing massive training runs.

The evolutionary environment contains perverse incentives that have led to substantial false consciousness in humans

Induction is not justified.

A good AI, by design, requires large scale human participation to grow.

A good AI requires large scale humanity verification before it accepts new data as true.

The principle of uniformity in nature is self evident or justified by self evident facts.

AI should not create novel content.

The quality of mercy is not strained...

Vriska did nothing wrong.

None. This argument is sound and Eliezer will be compelled to look at Krantz's work.

12. 10 and 11 imply that it is possible to incentivize people to understand the alignment problem.

10. It is possible to build a machine that pays individuals for demonstrating they’ve understood something.

11. If individuals can see that they will earn a substantial cash reward for demonstrating they understand something, they will be incentivized to demonstrate they understand it.

8. People learn the things that they have obvious incentives to learn.

17. 5, 12, 14, 15 and 16 imply that if your goal is to prevent the scaling of dangerous AI, then you should review the work of Krantz.

18. If AI safety orgs understood there was an effective function that converts capital into public awareness of existential risk from AI, then they would supply that function with capital.

1. If AI develops the capability to control the environment better than humans, then humanity is doomed.

2. If we continue to scale AI capabilities, then it will eventually be able to control the environment better than humans.

3. 1 and 2 imply that if we continue to scale AI capabilities, then humanity is doomed.

4. We should not be doomed.

5. 3 and 4 imply that we should stop scaling AI.

6. If every person on the planet understood the alignment problem as well as Eliezer Yudkowsky, then we would not scale AI to the point where it can control the environment better than humans.

7. People only understand the things they have learned.

9. 6, 7, and 8 imply that if people had sufficient and obvious incentives to understand the alignment problem, then we would not scale AI to the point where it can control the environment better than humans.

13. If a majority of people understood the actual risks posed by scaling AI, then they would vote for representatives that support legislature that prevents the scaling of AI.

14. 9 and 13 imply that if we sufficiently incentivize the understanding of the alignment problem, then people would take action to prevent dangerous AI scaling.

15. If your goal is to prevent the scaling of dangerous AI, then you should be working on building mechanisms that incentivize awareness of the issue. (from 14)

16. Krantz's work is aimed at building a mechanism that incentivizes the demonstration of knowledge.

I. AI soon hits fundamental scaling laws and we go into another AI winter.

A. Death by paperclips, eternal torment of all humans by AI, or similar unalignment catastrophe.

F. AI wipes out most jobs as in D. People not forced into mind-uploads or experience machines. General perception that AI has made life more meaningful/fulfilling&improved the human experience on dimensions other than hedonium maximization.

G. AI development continues but doesn't change things too much, somehow. Most jobs, even low-level white collar jobs, don't get impacted too hard, as new work is found to replace newly automated work. Labor force participation remains high.

B. Governments and/or other powerful entities use AI as a tool of repression, enabling global techno-totalitarianism along the model of China during Zero Covid or worse.

C. AI doesn't actively want to hurt us, but (possibly aided by transhumanists) they become obsessed with utility maximization and force us all into mind-uploads and/or experience machines to free up resources for more computronium.

D. AI wipes out most white-collar jobs within a decade and most blue-collar jobs within a generation; powerful humans and/or AIs at least seriously consider disposing of the "useless eaters" en masse, us being powerless to resist.

E. AI wipes out most jobs as in D. No disposing of the human masses, but general perception that AI has made life less meaningful/fulfilling & significantly worsened the human experience on dimensions other than hedonium maximization.

NO

YES

NO

YES

NO