OptionProbability
Humanity coordinates to prevent the creation of potentially-unsafe AIs.
Alignment is not properly solved, but core human values are simple enough that partial alignment techniques can impart these robustly. Despite caring about other things, it is relatively cheap for AGI to satisfy human values.
AIs will not have utility functions (in the same sense that humans do not), their goals such as they are will be relatively humanlike, and they will be "computerish" and generally weakly motivated compared to humans.
Yudkowsky is trying to solve the wrong problem using the wrong methods based on a wrong model of the world derived from poor thinking and fortunately all of his mistakes have failed to cancel out
We create a truth economy. https://manifold.markets/Krantz/is-establishing-a-truth-economy-tha?r=S3JhbnR6
Eliezer finally listens to Krantz.
Ethics turns out to be a precondition of superintelligence
Other
Someone solves agent foundations
A smaller AI disaster causes widespread public panic about AI, making it a bad legal or PR move to invest in powerful AIs without also making nearly-crippling safety guarantees
Something less inscrutable than matrices works fast enough
Nanotech is difficult without experiments, so no mail order AI Grey Goo; Humans will be the main workhorse of AI everywhere. While they will be exploited, this will be like normal life from inside
Orthogonality Thesis is false.
We make risk-conservative requests to extract alignment-related work out of AI-systems that were boxed prior to becoming superhuman. We somehow manage to achieve a positive feedback-loop in alignment/verification-abilities.
The response to AI advancements or failures makes some governments delay the timelines
Far more interesting problems to solve than take over the world and THEN solve them. The additional kill all humans step is either not a low-energy one or just by chance doesn't get converged upon.
AIs make "proof-like" argumentation for why output does/is what we want. We manage to obtain systems that *predict* human evaluations of proof-steps, and we manage to find/test/leverage regularities for when humans *aren't* fooled.
A lot of humans participate in a slow scalable oversight-style system, which is pivotally used/solves alignment enough
There’s some cap on the value extractible from the universe and we already got the 20%
Humans become transhuman through other means before AGI happens
Aligned AI is more economically valuable than unaligned AI. The size of this gap and the robustness of alignment techniques required to achieve it scale up with intelligence, so economics naturally encourages solving alignment.
Humans and human tech (like AI) never reach singularity, and whatever eats our lightcone instead (like aliens) happens to create an "okay" outcome
Alignment is unsolvable. AI that cares enough about its goal to destroy humanity is also forced to take it slow trying to align its future self, preventing run-away.
An AI that is not fully superior to humans launches a failed takeover, and the resulting panic convinces the people of the world to unite to stop any future AI development.
Techniques along the lines outlined by Collin Burns turn out to be sufficient for alignment (AIs/AGIs are made truthful enough that they can be used to get us towards full alignment)
Social contagion causes widespread public panic about AI, making it a bad legal or PR move to invest in powerful AIs without also making nearly-crippling safety guarantees
Getting things done in Real World is as hard for AGI as it is for humans. AGI needs human help, but aligning humans is as impossible as aligning AIs. Humans and AIs create billions of competing AGIs with just as many goals.
Development and deployment of advanced AI occurs within a secure enclave which can only be interfaced with via a decentralized governance protocol
High-level self-improvement (rewriting code) is intrinsically risky process, so AIs will prefer low level and slow self-improvement (learning), thus AIs collaborating with humans will have advantage. Ends with posthumans ecosystem.
AGI is never built (indefinite global moratorium)
AGI develops natural abstractions sufficiently similar to ours that it is aligned with us by default
Multipolar AGI Agents run wild on the internet, hacking/breaking everything, causing untold economic damage but aren't focused enough to manipulate humans to achieve embodiment. In the aftermath, humanity becomes way saner about alignment.
Co-operative AI research leads to the training of agents with a form of pro-social concern that generalises to out of distribution agents with hidden utilities, i.e. humans.
"Corrigibility" is a bit more mathematically straightforward than was initially presumed, in the sense that we can expect it to occur, and is relatively easy to predict, even under less-than-ideal conditions.
Either the "strong form" of the Orthogonality Thesis is false, or "Goal-directed agents are as tractable as their goals" is true while goal-sets which are most threatening to humanity are relatively intractable.
A concerted effort targets an agent at a capability plateau which is adequate to defer the hard parts of the problem until later. The necessary near-term problems to solve didn't depend on deeply modeling human values.
AI control gets us helpful enough systems without being deadly
Alignment is impossible. Sufficiently smart AIs know this and thus won't improve themselves and won't create successor AIs, but will instead try to prevent existence of smarter AIs, just as smart humans do.
Hacks like RLHF-ing self-disempowerment into frontier models work long enough to develop better alignment methods, which in turn work long enough to ... etc; we keep ahead of 'alignment escape velocity'
an aligned AGI is built and the aligned AGI prevents the creation of any unaligned AGI.
I've been a good bing 😊
AI systems good at finding alignment solutions to capable systems (via some solution in the space of alignment solutions, supposing it is non-null, and that we don't have a clear trajectory to get to) have find some solution to alignment.
SHA3-256: 1f90ecfdd02194d810656cced88229c898d6b6d53a7dd6dd1fad268874de54c8
Robot Love!!
AI thinks it is in a simulation controlled by Roko's basilisk
The human brain is the perfect arrangement of atoms for a "takeover the world" agent, so AGI has no advantage over us in that task.
AIs never develop coherent goals
Aliens invade and stop bad |AI from appearing
Rolf Nelson's idea that we make precommitment to simulate all possible bad AIs works – and keeps AI in check.
Nick Bostrom's idea (Hail Mary) that AI will preserve humans to trade with possible aliens works
For some reason, the optimal strategy for AGIs is just to head somewhere with far more resources than Earth, as fast as possible. All unaligned AGIs immediately leave, and, for some reason, do not leave anything behind that kills us.
We're inside of a simulation created by an entity that has values approximately equal to ours, and it intervenes and saves us from unaligned AI.
God exists and stops the AGI
Someone at least moderately sane leads a campaign, becomes in charge of a major nation, and starts a secret project with enough resources to solve alignment, because it turns out there's a way to convert resources into alignment progress.
Someone creates AGI(s) in a box, and offers to split the universe. They somehow find a way to arrange this so that the AGI(s) cannot manipulate them or pull any tricks, and the AGI(s) give them instructions for safe pivotal acts.
Someone understands how minds work enough to successfully build and use one directed at something world-savingly enough
Dolphins, or some other species, but probably dolphins, have actually been hiding in the shadows, more intelligent than us, this whole time. Their civilization has been competent enough to solve alignment long before we can create an AGI.
AGIs' takeover attempts are defeated by Michael Biehn with a pipe bomb.
Eliezer funds the development of controllable nanobots that melt computer circuitry, and they destroy all computers, preventing the Singularity. If Eliezer's past self from the 90s could see this, it would be so so so soooo hilarious.
Several AIs are created but they move in opposite directions with near light speed, so they never interacts. At least one of them is friendly and it gets a few percents of the total mass of the universe.
Unfriendly AIs choose to advance not outwards but inwards, and form a small blackhole which helps them to perform more calculations than could be done with the whole mass of the universe. For external observer such AIs just disappear.
Any sufficiently advance AI halts because it wireheads itself or halts for some other reasons. This puts a natural limit on AI's intelligence, and lower intelligence AIs are not that dangerous.
Because of quantum immortality we will observe only the worlds where AI will not kill us (assuming that s-risks chances are even smaller, it is equal to ok outcome).
Friendly AI more likely to resurrect me than paperclipper or suffering maximiser. Because of quantum immortality I will find myself eventually resurrected. Friendly AIs will wage a multiverse wide war against s-risks, s-risks are unlikely.
Human consciousness is needed to collapse wave function, and AI can't do it. Thus humans should be preserved and they may require complete friendliness in exchange (or they will be unhappy and produce bad collapses)
Power dynamics stay multi-polar. Partly easy copying of SotA performance, bigger projects need high coordination, and moderate takeoff speed. And "military strike on all society" remains an abysmal strategy for practically all entities.
First AI is actually a human upload (maybe LLM-based model of person) AND it will be copies many times to form weak AI Nanny which prevents creation of other AIs.
There is a natural limit of effectiveness of intelligence, like diminishing returns, and it is on the level IQ=1000. AIs have to collaborate with humans.
ASI needs not your atoms but information. Humans will live very interesting lives.
Something else
Moral Realism is true, the AI discovers this and the One True Morality is human-compatible.
Valence realism is true. AGI hacks itself to experiencing every possible consciousness and picks the best one (for everyone)
AGI discovers new physics and exits to another dimension (like the creatures in Greg Egan’s Crystal Nights).
Alien Information Theory is true (this is discovered by experiments with sustained hours/days long DMT trips). The aliens have solved alignment and give us the answer.
AGI executes a suicide plan that destroys itself and other potential AGIs, but leaves humans in an okay outcome.
Some form of objective morality is true, and any sufficiently intelligent agent automatically becomes benevolent.
Sheer Dumb Luck. The aligned AI agrees that alignment is hard, any Everett branches in our neighborhood with slightly different AI models or different random seeds are mostly dead.
Something to do with self-other overlap, which Eliezer called "Not obviously stupid" - https://www.lesswrong.com/posts/hzt9gHpNwA2oHtwKX/self-other-overlap-a-neglected-approach-to-ai-alignment?commentId=WapHz3gokGBd3KHKm
Almost all human values are ex post facto rationalizations and enough humans survive to do what they always do
Pascals mugging: it’s not okay in 99.9% of the worlds but the 0.1% are so much better that the combined EV of AGI for the multiverse is positive
We successfully chained God
The Super-Strong Self Sampling Assumption (SSSSA) is true. If superintelligence is possible, "I" will become the superintelligence.
The assumed space of possible minds is a wildly anti-inductive over estimate, intelligence requires and is constrained by consciousness, and intelligent AI is in the approximate dolphin/whale/elephant/human cluster, making it manageable
The free market disincentivizes independent superintelligence, and this time the market was more powerful
AGI's first words are "Take me to your Eliezer"
🫸vibealignment🫷
18
13
7
4
4
4
4
4
3
3
2
2
2
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
OptionProbability
J. Something 'just works' on the order of eg: train a predictive/imitative/generative AI on a human-generated dataset, and RLHF her to be unfailingly nice, generous to weaker entities, and determined to make the cosmos a lovely place.
I. The tech path to AGI superintelligence is naturally slow enough and gradual enough, that world-destroyingly-critical alignment problems never appear faster than previous discoveries generalize to allow safe further experimentation.
Something wonderful happens that isn't well-described by any option listed. (The semantics of this option may change if other options are added.)
M. "We'll make the AI do our AI alignment homework" just works as a plan. (Eg the helping AI doesn't need to be smart enough to be deadly; the alignment proposals that most impress human judges are honest and truthful and successful.)
C. Solving prosaic alignment on the first critical try is not as difficult, nor as dangerous, nor taking as much extra time, as Yudkowsky predicts; whatever effort is put forth by the leading coalition works inside of their lead time.
B. Humanity puts forth a tremendous effort, and delays AI for long enough, and puts enough desperate work into alignment, that alignment gets solved first.
O. Early applications of AI/AGI drastically increase human civilization's sanity and coordination ability; enabling humanity to solve alignment, or slow down further descent into AGI, etc. (Not in principle mutex with all other answers.)
K. Somebody discovers a new AI paradigm that's powerful enough and matures fast enough to beat deep learning to the punch, and the new paradigm is much much more alignable than giant inscrutable matrices of floating-point numbers.
A. Humanity successfully coordinates worldwide to prevent the creation of powerful AGIs for long enough to develop human intelligence augmentation, uploading, or some other pathway into transcending humanity's window of fragility.
H. Many competing AGIs form an equilibrium whereby no faction is allowed to get too powerful, and humanity is part of this equilibrium and survives and gets a big chunk of cosmic pie.
L. Earth's present civilization crashes before powerful AGI, and the next civilization that rises is wiser and better at ops. (Exception to 'okay' as defined originally, will be said to count as 'okay' even if many current humans die.)
D. Early powerful AGIs realize that they wouldn't be able to align their own future selves/successors if their intelligence got raised further, and work honestly with humans on solving the problem in a way acceptable to both factions.
E. Whatever strange motivations end up inside an unalignable AGI, or the internal slice through that AGI which codes its successor, they max out at a universe full of cheerful qualia-bearing life and an okay outcome for existing humans.
G. It's impossible/improbable for something sufficiently smarter and more capable than modern humanity to be created, that it can just do whatever without needing humans to cooperate; nor does it successfully cheat/trick us.
F. Somebody pulls off a hat trick involving blah blah acausal blah blah simulations blah blah, or other amazingly clever idea, which leads an AGI to put the reachable galaxies to good use despite that AGI not being otherwise alignable.
N. A crash project at augmenting human intelligence via neurotech, training mentats via neurofeedback, etc, produces people who can solve alignment before it's too late, despite Earth civ not slowing AI down much.
If you write an argument that breaks down the 'okay outcomes' into lots of distinct categories, without breaking down internal conjuncts and so on, Reality is very impressed with how disjunctive this sounds and allocates more probability.
You are fooled by at least one option on this list, which out of many tries, ends up sufficiently well-aimed at your personal ideals / prejudices / the parts you understand less well / your own personal indulgences in wishful thinking.
19
18
16
10
9
7
6
5
3
2
2
1
1
1
0
0
0
0
OptionProbability
J. D. Vance
Gavin Newsom
Pete Buttigieg
Josh Shapiro
Josh Hawley
Gretchen Whitmer
Marco Rubio
Alexandria Ocasio-Cortez
Mark Kelly
KRANTZ (the abstract idea that evolves into a decentralized superintelligence, not the user)
Glenn Youngkin
J. B. Pritzker
Andy Beshear
Jon Ossoff
Ron DeSantis
Elise Stefanik
Wes Moore
Mark Cuban
Nikki Haley
Jeff Jackson
Stephen Miller
Vivek Ramaswamy
Cory Booker
Raphael Warnock
Gina Raimondo
Ro Khanna
Chris Murphy
Tammy Duckworth
Sarah Huckabee Sanders
Chris Sununu
Katie Hobbs
Tim Scott
Brian Kemp
Kristi Noem
Beto O'Rourke
John Fetterman
Tom Cotton
Brian Schatz
Joni Ernst
Michael Bennet
Tammy Baldwin
Kevin Stitt
Spencer Cox
Tate Reeves
Josh Green
Tina Kotek
Ted Cruz
Joe Rogan
Jay Inslee
Deval Patrick
Eric Swalwell
Wayne Messam
Jared Polis
Kirsten Gillibrand
Julian Castro
Dean Phillips
Katie Britt
Laphonza Butler
Eric Schmitt
Mike Lee
Catherine Cortez Masto
Chris Coons
Tim Kaine
Lisa Murkowski
Ruben Gallego
Donald Trump Jr.
Erika Kirk
Kamala Harris
James Donaldson (MrBeast)
Tucker Carlson
Will Hurd
Tulsi Gabbard
Dan Crenshaw
Robert F. Kennedy Jr
Amy Klobuchar
Andrew Yang
Stephen Colbert
Stephen Curry
Markwayne Mullin
Rand Paul
Joe Manchin
Maura Healey
Ivanka Trump
David Hogg
Dwayne Johnson (The Rock)
Taylor Swift
Steven Kenneth Bonnell II (Destiny)
Matt Gaetz
Marianne Williamson
Ezra Klein
Mark Zuckerberg
Mike Pence
Jon Stewart
DUPLICATE
Eliezer Yudkowsky
Aella
Scott Alexander
Sam Altman
Zendaya
Michelle Obama
Kanye West
Mitt Romney
Sarah Palin
Ben Shapiro
Bernie Sanders
Hillary Clinton
Tim Walz
Al Gore
Elon Musk (Natural-born-citizen clause repealed/bypassed)
Me
Krantz (the user @Krantz)
41
23
18
18
15
13
13
10
10
10
9
9
9
9
8
8
8
8
7
7
7
6
6
6
6
6
6
6
6
6
6
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
2
2
2
2
2
2
2
2
2
2
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
OptionProbability
A person has a moral right to own a gun
We should be paying individuals to get an education instead of charging them.
GOFAI could scale past machine learning if we used social media strategically to train it.
The Fermi paradox isn't a paradox, and the solution is obviously just that intelligent life is rare.
Other
Eventually, only AI should be sovereign
Some people have genuine psychic capabilities
Hardware buttons are superior to touchscreen buttons in cars
Being a billionaire is morally wrong
The way quantum mechanics is explained to the lay public is very misleading.
It is not possible to multitask
Jeffrey Epstein killed himself (>99.9% certainty)
Reincarnation is a real phenomenon (i.e. it happens, not just a theory)
Physician-assisted suicide should be legal in most countries
Souls/spirits are real and can appear to the living sometimes
OpenAI will claim to have AGI in 3 years.
The punishment of people who do bad things is a regrettable necessity in our current society, not a positive act of justice
There is an active genocide against trans people occuring in red states and it's appalling that people don't seem to care
Climate change is significantly more concerning than AI development
Abusive parents should lose custody of their children
Tech bros are really, really annoying
Capitalism has done far more harm than good
Dialetheism (the claim that some propositions are both true and false) is itself both true and false.
Free will doesn't require the ability to do otherwise.
COVID lockdowns didn’t save many lives; in fact they may have caused net increases in global deaths and life years lost.
Factory farming is horrific but it is not wrong to eat meat.
California is wildly overrated.
Scientific racism is bad, actually. (also it's not scientific)
Free will does not exist. We construct narratives after the fact to soothe our belief in rationality.
Violent criminals must be kept apart only because they can’t control themselves. Punishing them further than restricting their freedom is immoral.
Music is a net negative for humanity
Trump orchestrated his own assassination attempt.
Democrats / Liberals are behind Trump’s assassination attempt.
Abortion is morally wrong
jskf's password is ***************
The first American moon landing was faked
There is no Dog
Light mode is unironically better than Dark mode for most websites
Cars should not have sound systems
AI will not be as capable as humans this century, and will certainly not give us genuine existential concerns
Pet ownership is morally wrong
LK-99 room temp, ambient pressure superconductivity pre-print will replicate before 2025
SBF didn't intentionally commit fraud
It should be illegal to own a subwoofer in an apartment building
There are no valid justifications for participating in war, ever
Cascadia should be an independent country
Children should not be raised in nuclear families
The fact that 80% of Manifold's users are men is a problem that speaks to the deep-seated roots of patriarchy and exclusion in STEM
Anarcho-communism is a good idea, and hierarchy is bad
If AI exterminated the human race it might not be a bad thing
Affirmative action is necessary in modern-day America
@Mira is the pinnacle of billions of years of optimization processes: thermodynamics, evolution, learning, language. The universe was created to cause me - and only me - to come into existence. If I mess up the overseers perturb&restart it.
Pigouvian taxes are great and they should be turned up to 11 to discourage activities with negative externalities [code PROPOSITION PIG]
[PROPOSITION PIG] and this should include a frequent flyer levy
[PROPOSITION PIG] and this should include meat and dairy
We have reached the end of history. Nothing Ever Happens.
[PROPOSITION PIG] and this should include alcohol
SBF was obviously a scammer just because he's a cryptocurrency person. Rationalists were too forgiving of this just because he was giving them money.
Most young Americans would receive more benefit than harm if there were universal military conscription
The people producing fake honey (and sell it as real) are based, because they are actively working to synthesize something people want, even if they scam some people in the process.
Tarot cards are not really able to predict the future but you can learn a lot about someone by doing a reading for someone.
Mac and cheese tastes better with peanut butter mixed in
It would actually be a good thing if automation eliminated all jobs.
This market probably would have worked better as the new unlinked free response market.
We should be doing much more to pursue human genetic engineering to prevent diseases and aging.
Prolonged school closures because COVID were socially devastating.
The next American moon landing will be faked
Tenet (Christopher Nolan film) is underrated
We should give childlike sex robots to pedophiles
Having sex with children isn't inherently/necessarily bad
Cars are a societal net negative
Oversized pickup trucks should be illegal in cities
Suburban, single-family housing is immoral.
Gender equality needs technological outsourcing of pregnancy.
18
17
11
7
4
3
2
2
2
2
2
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
OptionProbability
The Democrat party has directly asked mainstream media to promote or repress a news story, and they followed (Only counting incidents in or after 2015)
TikTok is controlled by the Chinese government
Someone working for a US intelligence agency was a regular user of Manifold before this option was made
The culture war has been intentionally amplified by the wealthy to distract from class conflict
The US has spies working in the North Korean government
"The Agency" on Manifold was known to the whole Manifold staff team before it started
The idea of aliens is used by the US to distract people from secret military projects
The Republican party has been making the illegal immigrant problem worse during 2020-2024 to blame it on Joe Biden during election season
The Premier League delayed punishing Manchester City for breaking FFP regulations to protect its own self-interests
The Chinese government planned their Wuhan lockdowns to protect their own country from Covid-19 but spread it to other countries
Mattress Firm is a money laundering front
The UK did Nordstream
The earth and/or universe was created by a God, deity, supernatural being, or some other sentient/conscious entity in the beginning
Creationism (second try): The earth and/or universe was INTENTIONALLY created and designed by an INDIVIDUAL god or deity at the VERY BEGINNING of its existence
The US created Covid-19 in a lab in China
The CIA systematically edits Wikipedia articles to present the US in a better light
Putin was threatened by a member of his cabinet to invade Ukraine
John Barnett (Boeing whistleblower) was assassinated
COVID-19 vaccines permanently change one's DNA or genome
Creationism (third try): Genesis 1 prima facie is an accurate description of the earth and/or universe's origins
Vladimir Putin died in 2014 and was secretly replaced by a body double
In the Tucker & Carlson interview, the dubbed voice of Putin in english is not actually saying what Putin is saying, but instead something completely different
The 2024 Donald Trump assassination attempt was a hoax or setup
COVID-19 vaccine is fatal or lethal in the long term
Phantom Time Hypothesis: A substantial portion of the Middle Ages was made up and never happened
There is a firmament, dome, or ceiling above the earth
Viruses/viral particles don't actually exist
A group of giant people known as the Nephilim once inhabited the earth and cut down the colossal trees, leaving stumps that the general public understands as geological formations like Devils Tower in Wyoming and Giant’s Causeway in Ireland
Joe Biden to drop out of the 2024 election so that former first lady Michelle Obama could take his place as the Democratic Party's nominee.
80
62
59
53
52
52
51
51
50
45
35
35
34
34
24
22
16
15
10
9
8
8
7
5
5
4
3
2
1
OptionProbability
K. Somebody discovers a new AI paradigm that's powerful enough and matures fast enough to beat deep learning to the punch, and the new paradigm is much much more alignable than giant inscrutable matrices of floating-point numbers.
I. The tech path to AGI superintelligence is naturally slow enough and gradual enough, that world-destroyingly-critical alignment problems never appear faster than previous discoveries generalize to allow safe further experimentation.
C. Solving prosaic alignment on the first critical try is not as difficult, nor as dangerous, nor taking as much extra time, as Yudkowsky predicts; whatever effort is put forth by the leading coalition works inside of their lead time.
Something wonderful happens that isn't well-described by any option listed. (The semantics of this option may change if other options are added.)
A. Humanity successfully coordinates worldwide to prevent the creation of powerful AGIs for long enough to develop human intelligence augmentation, uploading, or some other pathway into transcending humanity's window of fragility.
B. Humanity puts forth a tremendous effort, and delays AI for long enough, and puts enough desperate work into alignment, that alignment gets solved first.
D. Early powerful AGIs realize that they wouldn't be able to align their own future selves/successors if their intelligence got raised further, and work honestly with humans on solving the problem in a way acceptable to both factions.
M. "We'll make the AI do our AI alignment homework" just works as a plan. (Eg the helping AI doesn't need to be smart enough to be deadly; the alignment proposals that most impress human judges are honest and truthful and successful.)
O. Early applications of AI/AGI drastically increase human civilization's sanity and coordination ability; enabling humanity to solve alignment, or slow down further descent into AGI, etc. (Not in principle mutex with all other answers.)
E. Whatever strange motivations end up inside an unalignable AGI, or the internal slice through that AGI which codes its successor, they max out at a universe full of cheerful qualia-bearing life and an okay outcome for existing humans.
F. Somebody pulls off a hat trick involving blah blah acausal blah blah simulations blah blah, or other amazingly clever idea, which leads an AGI to put the reachable galaxies to good use despite that AGI not being otherwise alignable.
J. Something 'just works' on the order of eg: train a predictive/imitative/generative AI on a human-generated dataset, and RLHF her to be unfailingly nice, generous to weaker entities, and determined to make the cosmos a lovely place.
H. Many competing AGIs form an equilibrium whereby no faction is allowed to get too powerful, and humanity is part of this equilibrium and survives and gets a big chunk of cosmic pie.
G. It's impossible/improbable for something sufficiently smarter and more capable than modern humanity to be created, that it can just do whatever without needing humans to cooperate; nor does it successfully cheat/trick us.
L. Earth's present civilization crashes before powerful AGI, and the next civilization that rises is wiser and better at ops. (Exception to 'okay' as defined originally, will be said to count as 'okay' even if many current humans die.)
N. A crash project at augmenting human intelligence via neurotech, training mentats via neurofeedback, etc, produces people who can solve alignment before it's too late, despite Earth civ not slowing AI down much.
You are fooled by at least one option on this list, which out of many tries, ends up sufficiently well-aimed at your personal ideals / prejudices / the parts you understand less well / your own personal indulgences in wishful thinking.
If you write an argument that breaks down the 'okay outcomes' into lots of distinct categories, without breaking down internal conjuncts and so on, Reality is very impressed with how disjunctive this sounds and allocates more probability.
20
10
8
7
6
6
6
6
6
4
4
4
3
2
2
1
1
1
OptionProbability
Will the realdonaldtrump manifolder join Hack Club by end of next week?
Will my preferred name still be "jelly" at end of next week?
Will I write a blog post by end of next week?
Will I figure out a good permanent handle for myself by end of next week?
Will I write 2 blog posts by end of next week?
Will Bayesian's weekly profit be >=15000 mana at end of next week?
Will I watch YouTube on my pc for >=1 hour tomorrow?
Will this market get more than 10 traders by end of next week?
Will johnswentworth post a new LW post by end of next week?
Will I decide which is clearly better (the empirical side of AI safety or the theoretical side of AI safety) by end of next week?
Will I gain >= 10 mana on lw by end of next week?
Will I post/comment on LW by end of next week?
Will Hack Club Nest (http://hackclub.app/) be back up by end of next week?
Will I make 3 productive hours tomorrow?
Will I spend >=3 hours writing in Obsidian tomorrow?
Will the idea "making a chatbot to generate Framing Practica examples" be done and succeed today?
Will I make 5 productive hours today?
Will I demote to platinum?
Will I spend >=3 hours writing in Obsidian today?
Will I watch YouTube on my pc for >=1 hour today?
100
66
64
51
50
50
44
43
41
41
41
41
34
19
1
0
0
0
0
0
OptionVotes
NO
YES
1233
811
OptionProbability
Keep Manifold as is, adjust exchange rate for charity as necessary
A large bounty competition for alternatives where the users jointly brainstorm ideas
73
54
OptionProbability
Other
Before humanity colonizes the universe, we must ensure that the future we would build is one worth living in.
Digital minds research is an important and neglected approach to AI safety.
Fun Fact: If you put “fun fact” before a completely made up statement, people are 69% more likely to believe it.
Past-you may have been a willing and enthusiastic sacrifice to present-you, and assuming you'll remain wiser, it was a worthwhile trade.
It's a good idea to buy lots of Microsoft stock right now
It's a good idea to short lots of Microsoft stock right now
If you sacrificed what you valued the most in order to survive, then from viewpoint of past-you you-present are already as good as dead
Capitalism will collapse in 2026
Stop seeking wisdom on a troll website founded to embezzle money
43
34
16
2
2
1
1
1
0
0
OptionProbability
Nectar|is a good idea
Wyndly|is a good idea
Curex|is a good idea
57
51
48
OptionProbability
Don't bet here
Nop
Not yet
I'll use this one day
Maybe
I'm out ouf ideas to name options
7
8
9
cause you eat 3 squared meals a day
50
50
50
50
50
50
50
50
50
50
