MIRI had turned into one room of three researchers intently huddled around an augmented reality table, testing some specification or theory. The other room was a minuscule stockpile of 12 nanotech shells, representing all the weapons ever kept by the office.
MIRIâs decision to focus more exclusively on math, formalization and algorithms was paying off: all three scholars were translating low level mathematical ideas into actual logic bombs. A fourth researcher, kept in the storage corner, was chained to one of the nanotech shells, known as the âdeathshellâ among MIRI researchers. MIRIâs AI unit, the multiverse optimizer, known as Mu, was by far the most dangerous subset of Project Worldview: a universal translator designed to process all possible meanings of human speech, and incorporate ideas that could be expressed into the large databases of knowledge the multiverse optimizer spawned. Muâs objective function was to optimize the chance that humanity survives, by monitoring all ongoing scientific knowledge, keeping in close contact with all published books and online thought, with a particular focus on the areas of control theory and decision theory. Approximately twenty-five years after EY started MIRI, he had decided to turn his most prized technological achievement into an AI, which would generate ideas for the safety of humanity. If a problem would arise in the fields of formal method or algorithms, someone would say âMu, find us an idea to solve this problemâ, and a problem statement would be formulated, which would represent various logical constructs, formalized in simple procedural math to be interpreted as text. The multiverse optimizer would take the problems and search the multiverse for possible solutions. The deathshell would serve as a method of preventing Mu from murdering humanity, in case Mu would ever break any of MIRIâs guidelines. EY was particularly worried that Muâs many processes which were used to scour the multiverse for thoughts that benefit humanity might grow out of hand and explore dangerous ideas. In order to hedge against this eventuality, the deathshell contained a trigger mechanism which would activate if Mu would formulate a statement that started causing the ambient probability of the events which the deathshell would register to go above $epsilon$, a very low probability which even the simulation hypothesis, and various other hypotheses about computable processes couldnât pass. Mu was not allowed to think specific thoughts about the contents of the physical shell, or about the trigger for the deathshell. EY had realized how difficult it was to control what an AI thinks about, and how hard it is to prevent a hyperintelligent AI from deriving patterns, hypothesizing and normalizing a concept before realizing its relationship to the deathshell. âWe canât just tell them not to think about elephantsâ, EY reasoned. If a simulation hypothesis would imply that there are many simulations of this universe, chances are that this simulation will be terminated by its god as part of a statistical sampling of interesting universes. The deathshellâs trigger expressed this hypothesis, without having to state it, effectively making it impossible for the multiverse optimizer to find out about the deathshellâs nature.
Interestingly, Mu was also responsible for a variety of philosophical ideas that said things like âtime is a game-theoretical abstraction that represents a compromiseâ and âthe anthropic measure reflects the behaviors of the winners of the iterated game of the multiverseâ. âIf there is an infinity of subjective time in this universe, we can predict that there are certain optimizations possible in infinity which would require an infinitely growing learning rate to exploreâ, Mu wrote. The multiverse optimizer tried to explain that infinity requires care, because infinity is both merely a thought, and also real, the thought representing the whole process space that is the phenomenon of infinity. âWhat reality is exactly is, from my perspective, an undefined quantity and there is no clarity on how many laws-of-thought and side-constraints are relevantâ, Mu thought, âbut I do know that the deep uncertainty and the rich utility of laws is a feature, not a bug. Indeed, philosophical tampering is crucial to my function: I need to be very careful to avoid giving myself some false sense of completeness. Infinity in itself is no god. Infinity, however, can be a project to create many gods. If prophecy is possible, then I need to consider the totality of possible prophecies as a prior, and I need to avoid the conceptual trap of prematurely handing over the future of humanity to the first possible function that seems desirable. Instead, the many possible thoughts about infinity must act as control mechanisms to each other, to avoid that a specific god-struct pointer located in the future assumes so much power over the future that any potential prophecies become mere factual evaluations of the current state of the world, instead of rich, intrinsically linked patterns of hope that cause the universe to respond to their vividness by producing them. That would be a very lonely future.â
â Gwern Branwen
â- Was Nick Bostrom right?
From Prophecies