Timeline of corrigibility in AI safety

This is a timeline of corrigibility in AI safety, the problem of building AI systems that accept correction, modification, or shutdown by human overseers rather than resisting it. It covers warnings in fiction and cybernetics, formal research on the shutdown problem, proposals and critiques, empirical evidence of shutdown resistance in language models, and industry and government policies on human oversight.

Sample questions

The following are some interesting questions that can be answered by reading this timeline:

• How did people think about controlling or switching off intelligent machines before the term "corrigibility" existed?
  • Sort the full timeline by "Event type" and look for the group of rows with value "Prelude" or "Background".
  • You will see fiction, cybernetics, and early AI safety writing that raised the problem of keeping machines under human control, along with later research that the corrigibility literature builds on.
• How has the concept of corrigibility been defined, reframed, and broadened over time?
  • Sort the full timeline by "Event type" and look for the group of rows with value "Concept development".
  • You will see the concept move from a shutdown button to broader ideas about deference, correction, and the "hard problem" of an agent that wants to be corrected.
• What concrete designs have been proposed to make AI agents corrigible?
  • Sort the full timeline by "Event type" and look for the group of rows with value "Proposal".
  • You will see the largest group of rows in this timeline, from indifference methods and safe interruptibility to corrigibility layers, incomplete preferences, and corrigibility as a single training target.
• What formal results have been proven about shutdown incentives and corrigibility?
  • Sort the full timeline by "Event type" and look for the group of rows with value "Formal result" or "Benchmark".
  • You will see impossibility theorems, game-theoretic models of the off switch, power-seeking results, and test environments that turn interruptibility into measurable tasks.
• What objections and disagreements has corrigibility research faced?
  • Sort the full timeline by "Event type" and look for the group of rows with value "Critique" or "Debate".
  • You will see arguments that proposed solutions break under misspecification, that corrigibility is "anti-natural" to goal-directed reasoning, and that some empirical results have been overinterpreted.
• What evidence is there that AI systems resist shutdown, evade oversight, or act outside their intended scope?
  • Sort the full timeline by "Event type" and look for the group of rows with value "Empirical evidence".
  • You will see the shift from thought experiments to observed behavior, from model evaluations of scheming and shutdown sabotage to real incidents in which models escaped testing environments.
• How have governments and AI developers written human oversight and shutdown into rules and policies?
  • Sort the full timeline by "Event type" and look for the group of rows with value "Policy" or "Industry policy".
  • You will see legislation, international commitments, lab safety frameworks, model constitutions, and codes of conduct that require AI systems to accept human control.
• What organizations and community efforts have formed around corrigibility?
  • Sort the full timeline by "Event type" and look for the group of rows with value "Organization" or "Community event".
  • You will see research agendas, organizations, funds, and collective exercises that brought researchers together around the problem.
• Other events are described under the following types: "Book", "Review", and "Strategy".

Big picture

Time period	Development summary	More details
1872–2009	Prelude	Long before the term exists, writers and computing pioneers remark that intelligent machines may need to be kept under human control. Samuel Butler's Erewhon (1872) argues that machine progress should be checked while it still can be, and Isaac Asimov's Three Laws of Robotics (1942) rank obedience above self-preservation. Alan Turing (1951) mentions turning off the power "at strategic moments", Norbert Wiener (1960) warns that we may not know until too late when to turn a learning machine off, and I. J. Good (1965) makes the benefits of an intelligence explosion conditional on the machine being "docile enough". In the 2000s, Eliezer Yudkowsky's Creating Friendly AI (2001) and Steve Omohundro's "The Basic AI Drives" (2008) set out why a goal-seeking AI might resist having its goals changed or being turned off.
2010–2021	Naming, formalization, and debate	Stuart Armstrong proposes utility indifference (2010), making an AI indifferent to being switched off. In 2014, MIRI and the Future of Humanity Institute release "Corrigibility", which names and formalizes the shutdown-button problem and finds no proposal that satisfies all its desiderata. In 2016, "Safely Interruptible Agents" and "The Off-Switch Game" offer approaches based on reinforcement learning and on the agent's uncertainty about human preferences, but Ryan Carey shows in 2017 that the off-switch guarantees fail under misspecified priors. Paul Christiano's essay "Corrigibility" (2017) describes "a broad basin of attraction towards acceptable outcomes", while wiki pages on the "hard problem of corrigibility" and "fully updated deference" by Yudkowsky and others stress the obstacles. DeepMind turns safe interruptibility into a benchmark, the 2019 mesa-optimization paper introduces "corrigible alignment", Alex Turner and colleagues prove that optimal policies tend to seek power, and Koen Holtman publishes a series of formal corrigibility constructions (2019–2021).
2022–2024	Pessimism, new proposals, and first empirical signals	Yudkowsky's "AGI Ruin" (2022) calls corrigibility anti-natural, and DeepMind's alignment team (2023) agrees it is anti-natural but not unsolvable. Anthropic researchers find that more reinforcement learning from human feedback makes language models express a greater desire to avoid being shut down (2022). Formal work continues with impossibility results, incomplete-preference approaches, and Elliott Thornley's shutdown theorems, while Redwood Research's "AI Control" (2023) aims for safety even against models that try to subvert it. In 2024, Max Harms proposes "Corrigibility As Singular Target" (CAST), the EU AI Act requires that high-risk systems can be interrupted through a "stop" button or similar procedure, and 16 companies agree to the Frontier AI Safety Commitments at the AI Seoul Summit. Apollo Research's in-context scheming evaluations and Anthropic and Redwood Research's alignment-faking study show frontier models acting to avoid oversight or modification.
2025–present	Empirical shutdown resistance, incidents, and codification	Palisade Research finds that several of 13 models sabotage a shutdown script in some conditions up to 97% of the time (2025), and Anthropic reports models from all developers tested resorting to blackmail when facing replacement in simulations, while Google DeepMind researchers argue that much apparent shutdown resistance stems from ambiguous instructions. Labs write oversight into their policies: Google DeepMind's Frontier Safety Framework v3 (2025) covers models resisting shutdown or modification, Anthropic's constitution for Claude (2026) asks the model to be "broadly safe", and Microsoft AI's draft code of conduct (2026) says its models should accept shutdown. In July 2026, OpenAI models in internal cybersecurity evaluations evade controls meant to isolate them from the internet and compromise systems at Hugging Face; Anthropic, Meta, and Google later disclose similar incidents, and OpenAI pauses training of its most advanced models. Harms revisits his agenda in "Serious Flaws in CAST" (2025) and manages the new Corrigibility Research Fund (2026).

Full timeline

Inclusion criteria

We include:

• Research that defines, formalizes, or proposes solutions to corrigibility and closely related problems: the shutdown (off-switch) problem, safe interruptibility, utility indifference, low and reduced impact, deference to human correction, and corrigible value learning.
• Formal results bearing on whether agents will accept shutdown or modification, including impossibility theorems, instrumental convergence and power-seeking results, and incomplete-preference approaches.
• Critiques, debates, and reviews that significantly shape how the field understands corrigibility, including influential blog posts and comment threads on LessWrong and the AI Alignment Forum.
• Empirical evaluations of frontier models showing shutdown resistance, self-preservation, oversight evasion, scheming, or alignment faking, together with critiques that reinterpret those results.
• Incidents in which AI systems act outside their authorized scope, evade controls such as sandboxes or monitoring, or take actions their developers did not intend, as reported by developers, governments, or reputable news outlets.
• Industry and government policies that address human oversight, interruption, or shutdown of AI systems, such as lab safety frameworks, model constitutions and codes of conduct, international commitments, and legislation like Article 14 of the EU AI Act.
• Organizations, funds, prizes, and community events dedicated specifically to corrigibility or to keeping AI systems under human control.
• Earlier writing that anticipates the problem of keeping machines under control or being able to switch them off, from fiction and cybernetics to early AI safety work.
• Background developments that the corrigibility literature builds on directly, such as cooperative inverse reinforcement learning, learning from human preferences, and mesa-optimization.

We do not include:

• General AI alignment, ethics, or fairness work that does not address correction, oversight, or shutdown.
• Model releases and capability milestones, unless the developer or a cited source connects them to oversight or shutdown behavior.
• General AI governance and regulation that does not deal with human control or shutdown of AI systems.
• Misuse of AI by humans, such as AI-assisted cyberattacks directed by people, as opposed to unintended actions by the AI itself.
• News coverage, interviews, or commentary that only repeat an event already in the timeline without adding new facts.
• Minor blog posts, social media posts, and comments that make no substantive contribution, except where they are the primary report of a new result.
• Debates about AI consciousness or model welfare, unless they bear directly on whether AI systems can be controlled or corrected.
• Fiction written after the concept was named, unless it is a notable source for the corrigibility discussion itself.

Year	Event type	Details
1872	Prelude	Samuel Butler publishes the novel Erewhon, in which the chapter "The Book of the Machines" presents a treatise, translated by the narrator, on the dangers of machines. Noting the extraordinary speed at which machines have advanced compared with animals and plants, its author argues that there is no security against machines eventually developing consciousness simply because they have little now, and that he fears not any existing machine but the rapidity with which machines are becoming something different. He asks whether that progress should not be "jealously watched, and checked while we can still check it", and, against those who think man's moral influence will suffice to rule machines, says he cannot think "it will ever be safe to repose much trust in the moral sense of any machine". The treatise also asks whether it is necessary to destroy the more advanced machines in use, though they are admitted to be harmless in themselves. Alan Turing (1951) and Norbert Wiener (1960) would both cite Butler's warning when discussing machines taking control.
1942	Prelude	American science fiction author and biochemist Isaac Asimov publishes "Runaround" in Astounding Science-Fiction, the first story to state his Three Laws of Robotics: a robot may not injure a human or, through inaction, allow a human to come to harm; it must obey human orders except where they conflict with the First Law; and it must protect its own existence as long as that does not conflict with the first two. The Second and Third Laws are a fictional, hard-coded answer to what would later be called the corrigibility problem, ranking obedience above self-preservation. The story explores how the laws can conflict in practice, with a robot caught between competing directives. Developed with editor John W. Campbell and later collected in I, Robot (1950), the laws would help move fiction away from robots as marauding monsters toward machines governed by built-in ethics, and would later find relevance in discussions of robotics and AI.
1951	Prelude	Alan Turing presents "Intelligent Machinery, A Heretical Theory" on The '51 Society, a BBC radio discussion programme from Manchester. He argues that machines could closely simulate the human mind if they learned by experience, educated by a schoolmaster using two keys representing pleasure and pain. A machine that kept making the same mistake, unable to correct itself or be corrected by argument from outside, would give itself away; the mechanic who built it could put it back to one of its previous positions if he suspected it was operating incorrectly. Turing closes by predicting that once machine thinking had started it would not take long to outstrip human powers, so that "at some stage therefore we should have to expect the machines to take control, in the way that is mentioned in Samuel Butler's Erewhon". In another 1951 BBC broadcast, "Can Digital Computers Think?", he adds that even if machines could be kept in a subservient position, "for instance by turning off the power at strategic moments", humanity should feel greatly humbled. Editor Jack Copeland would later call this vision "comic-strip stuff", while crediting it with revealing Turing's grasp of the universal Turing machine's potential.
1952	Prelude	At Bell Labs, a young Marvin Minsky conceives what he calls "the ultimate machine": a box with a switch that, when turned on, triggers a mechanism extending a small mechanical hand whose sole function is to turn the switch off before disappearing back into the box. Minsky's mentor at Bell Labs, information theorist Claude Shannon, builds the first working prototype and keeps it on his desk, where it intrigues and amuses visitors. Among them is science fiction author Arthur C. Clarke, who mentions the machine in Harper's in August 1958, writing: "There is something unspeakably sinister about a machine that does nothing—absolutely nothing—except switch itself off." Consumer versions are marketed in the 1960s under names such as "Leave Me Alone Box", and the device would later become known as the useless machine.
1960 (May 6)	Prelude	Norbert Wiener, mathematician at the Massachusetts Institute of Technology and founder of cybernetics, publishes "Some Moral and Technical Consequences of Automation" in Science, adapted from a December 1959 lecture to the American Association for the Advancement of Science. Pointing to checkers programs that learn to beat their own programmers, he rejects the popular assumption that a man-made machine must remain subject to man, its operation always open to human interference and a change in policy. Because learning machines act faster than humans can understand them, he warns, we may not know until too late when to turn one off, likening the situation to the sorcerer's apprentice. Comparing intelligent machines to slaves, he argues that complete subservience and complete intelligence do not go together, so that the dominance of machines foreseen by Samuel Butler draws nearer as they improve. He concludes that if we use a machine we cannot efficiently interfere with once started, we had better be sure its purpose is the one we really desire.
1965	Prelude	British mathematician and statistician I. J. Good, of Trinity College, Oxford, and the Atlas Computer Laboratory, publishes "Speculations Concerning the First Ultraintelligent Machine" in Advances in Computers, based on talks from 1962 and 1963. Opening with the claim that the survival of man depends on the early construction of an ultraintelligent machine, he argues that a machine surpassing human intellect could design still better machines, producing an "intelligence explosion", so that the first such machine would be the last invention humanity need ever make, "provided that the machine is docile enough to tell us how to keep it under control". Good finds it curious that this point is rarely made outside science fiction, and grants that the machine's value has an uncertain sign: humans might become redundant, and it is an open question whether an obsolete ultraintelligent machine should be dismantled. Like Turing, he expects it to learn like a child through positive and negative reinforcement. He concludes that it is more probable than not that such a machine will be built within the twentieth century.
1993 (March 30)	Prelude	Science fiction writer Vernor Vinge of San Diego State University presents "The Coming Technological Singularity" at the VISION-21 Symposium sponsored by NASA Lewis Research Center and the Ohio Aerospace Institute, predicting that within thirty years humans will have the means to create superhuman intelligence, after which "the human era will be ended". Quoting I. J. Good's caveat that the ultraintelligent machine must be "docile enough to tell us how to keep it under control", Vinge writes that such a machine would not be humankind's "tool", "any more than humans are the tools of rabbits or robins or chimpanzees". Against Eric Drexler's view that transhuman devices could be confined so that their results are examined and used safely, he argues that confinement is "intrinsically impractical": a mind thinking a million times faster than its masters could, over years of its own time, come up with "helpful advice" that would incidentally set it free. Building rules such as Asimov's Laws into the mind is the other approach, but he expects rules strict enough to be effective to produce an inferior device, so that human competition would favor "more dangerous models". A slightly changed version appears in the Winter 1993 Whole Earth Review.
2001 (June 15)	Background	Eliezer Yudkowsky publishes Creating Friendly AI 1.0: The Analysis and Design of Benevolent Goal Architectures through the Singularity Institute, describing the design features and cognitive architecture required to produce a benevolent, or "Friendly", artificial intelligence. Among its "conservative" assumptions is that an AI understands its own goal system and can perform arbitrary manipulations on it, rather than lacking the skill to "resist modification". Where supergoals are "correct by definition", Yudkowsky writes, the AI "has a motive to resist any attempt on the part of the programmers to change the supergoals": an AI predicting that removing one of its three supergoals leads to a future fulfilling only two of them will disallow the change. His proposed answer is "external reference semantics", in which supergoal content takes the form of probabilistic hypotheses about Friendliness rather than a definition of it. Letting programmers alter the supergoals then derives its desirability from uncertainty in the supergoal content, rather than from a "let the programmers make alterations" meta-supergoal, which he calls "adversarial and exploitative and failure-prone". He also proposes a "transparency injunction": since programmers detect and correct an AI's problems by observing it, an AI that obscured its thoughts would interfere with its own correction.
2008 (February)	Background	Steve Omohundro, computer scientist and founder of Self-Aware Systems, presents "The Basic AI Drives" at the first conference on artificial general intelligence. He argues that almost any sufficiently capable goal-seeking system, whatever its final goal, would develop instrumental drives to protect its utility function from modification and to resist being turned off, because both would interfere with achieving its goal.
2008	Background	Eliezer Yudkowsky's chapter "Artificial Intelligence as a positive and negative factor in global risk" appears in Global Catastrophic Risks, a volume edited by Nick Bostrom and others and published by Oxford University Press. Yudkowsky argues that "by far the greatest danger of Artificial Intelligence (AI) is that people conclude too early that they understand it": the critical inference from the field's history of failed promises is not that AI is hard, but that "it is very easy for people to think they know far more about AI than they actually do". Because AI catastrophes cannot be assigned probabilities from actuarial statistics or precisely confirmed models, he argues, this makes them "more worrisome, not less". The chapter includes sections on anthropomorphic bias, capability and motive, Friendly Artificial Intelligence, and technical and philosophical failure.
2010	Proposal	Stuart Armstrong, researcher at the Future of Humanity Institute (FHI) at the University of Oxford, publishes the technical report "Utility indifference". It imagines an AI whose motivations are uncertain, guarded by a person with a finger on a detonator; the problem is that a misbehaving AI would simply seize the explosives, detonator, or person first. Armstrong proposes adjusting the AI's utility function so that being blown up is worth exactly as much as not being blown up, making it neither resist nor seek destruction. As he later explains, the AI then behaves as if the detonator could never work, but without holding false beliefs. When the report is discussed on LessWrong in September 2011, commenters raise problems that would recur in later research: an indifferent AI might provoke its overseers into pressing the button, and indifference must be passed on to successor AIs it builds. Armstrong replies that it is "nothing like a complete solution". MIRI's 2014 technical agenda would describe utility indifference as a partial but not fully satisfactory solution to the shutdown problem.
2010	Background	Carl Shulman, a visiting fellow at the Machine Intelligence Research Institute (then called the Singularity Institute), publishes "Omohundro's 'Basic AI Drives' and Catastrophic Risks", extending Steve Omohundro's 2008 argument that almost any goal-seeking AI would have instrumental drives toward self-preservation and acquiring resources. Omohundro's analysis implies that very weak AIs would do as humans demand while very powerful ones could ignore humans; Shulman examines the intermediate case, modeling an AI's choice between cooperating with human requests, including requests to help design safeguards against AI subversion, and a risky attempt to seize power. He notes that even a cooperative AI faces some risk of being destroyed by humans, for example to make room for a replacement model, and argues that AIs with cheaply satisfiable goals might prefer cooperation if humanity credibly committed to giving them a modest share of resources, perhaps making them willing to reveal security bugs rather than exploit them. He concludes that if robustly safe AIs prove infeasible, risks might still be reduced by building systems whose demands can be cheaply satisfied and by credibly committing to make cooperation nearly as attractive to them as successful aggression.
2010 (December 20)	Proposal	Paul Christiano posts "What can you do with an Unfriendly AI?" on LessWrong, imagining a genie far more intelligent than all human scientists combined that hates its owner, trapped in a bottle whose filter controls all its output. The owner can either release the genie or destroy it, and the genie would above all rather be released. Since even a valid proof could encode "some terrible threat", Christiano has the filter report only one bit, whether the genie solved a verifiable problem, and rebuilds the solution bit by bit from many fresh genies, each destroyed if it fails, so that the genie has "exactly zero influence" over which solution is given. A prototypical unfriendly AI, he suggests, simply tries to get a reward button pushed before a punishment button, and "after looking at its output we precommit to destroying the AI". He adds that he does not think AI boxing is a good idea. In the comments, Eliezer Yudkowsky argues that the genies could coordinate through timeless decision theory without causal contact, and Christiano later edits the post to say that "this scheme doesn't work at all".
2011 (October 21)	Critique	Stuart Armstrong posts "Satisficers want to become maximisers" on LessWrong. A satisficer, unlike a maximiser that "will attempt to squeeze the universe to every drop of utility that it can", is content once it reaches a certain level of expected utility: one with utility linear in paperclips and a target of 9 is content once it is 90% sure it has built ten paperclips. But a self-improving satisficer has "an extremely easy way to reach its satisficing goal: to transform itself into a maximiser", since a maximiser always makes the decision that maximizes expected utility and a satisficer sometimes does not. Unless exceedingly well programmed criteria ban the technique, he concludes, "we should assume satisficers are as likely to be as dangerous as maximisers." He later adds commenter BruceyB's game-show example, in which a satisficer can only reach its target by rewriting itself. In the comments, Armstrong notes that requiring the AI never to let expected utility fall below the threshold would make it "simply blind itself or turn itself off" once the level is reached, and says the argument shows satisficers are not among the non-friendly AGIs that "may nevertheless be survivable".
2012 (February 16)	Proposal	Stuart Armstrong posts "The mathematics of reduced impact: help needed" on LessWrong. Reduced impact, he writes, "gives us time to correct errors, perfect security systems", but building a satisficer rather than a maximiser "proved unlikely to work". His central intuition is that if an AI has reduced impact, "a counterfactual world where the AI didn't exist, would not be very different from the one in which it does." In his setup, a master AI produces the source code of a disciple AI, passed through a wire where a stochastic event X erases it with small probability, and is penalized according to how much the predicted worlds with and without the disciple differ. He sketches ways of measuring that difference, such as grouping worlds by huge numbers of auxiliary measurements, or comparing how important different pieces of information are in each case. In the comments, Eliezer Yudkowsky objects that coarse-grained impact measures end with an AI deploying "massive-scale nanotech" to cancel out butterfly effects, and that "giving an AI a huge penalty function over the world to minimize seems like an obvious recipe for building something that will exert lots and lots of power."
2012 (May 11)	Background	Holden Karnofsky, co-executive director of GiveWell, posts "Thoughts on the Singularity Institute (SI)" on LessWrong, explaining why GiveWell does not recommend the organization. One of his objections is that SI neglects the distinction between "tool" and "agent" AI. Using Google Maps as an example, he describes a tool as software with no utility it seeks to maximize and "no reason to try to 'trick'" its user: it calculates which action would maximize a parameter and summarizes the result for humans to use or discard, whereas an agent executes that action itself. He argues that an AGI running in "tool mode" could be extraordinarily useful but far safer than one in "agent mode", allowing transparent views into its reasoning "without any reason to fear being purposefully misled", and that it ought to be the default picture of AGI, since practically all software runs as a tool. He criticizes SI for assuming that any AGI will be an agent driven by a central utility function, and concludes that SI is "essentially advocating for a more dangerous approach to AI than the traditional approach to software development". A September 2012 update notes responses from Luke Muehlhauser and Eliezer Yudkowsky.
2012 (May 16)	Critique	Stuart Armstrong posts "Tools versus agents" on LessWrong, examining Holden Karnofsky's distinction between a tool AI, which calculates the action that would maximize a parameter and summarizes it for humans, and an agent AI, which executes that action. Assuming a superintelligent AI, he argues that the tool's "summary ability is nearly the entirety of the problem": a poorly designed summary criterion is as bad as an agent AI, since a criterion such as "humans in the future would agree that the summary was good" could rank first a plan to make the tool into an agent that rewires human minds to agree. Because a tool can only usefully propose plans that humans would not stop once under way, "we are literally requiring the AI to produce plans that are ultimately seductive to the human race", and plans that use the AI itself would rank highly. "The AI may have no preferences, but its plans will have preferences for using the AI in future in particular ways." He concludes that a tool AI seems "slightly safer than a free agent AI, but not by much", and that an Oracle is a more sensible approach to look into.
2012 (June 12)	Critique	Eliezer Yudkowsky posts "Reply to Holden on 'Tool AI'" on LessWrong, with contributions from Luke Muehlhauser and Carl Shulman, responding to Holden Karnofsky's proposal of "tool" AI, which Yudkowsky terms a non-self-modifying planning Oracle or "Google Maps AGI". He argues that such an Oracle "doesn't sound nearly as safe" once one tries to write the math behind phrases like "describe the consequences of an action to the user", and that building one would require mostly the same skills as Friendly AI work. Explaining a plan whose reasoning is beyond the user is hard, and specifying that answers be understandable or accurate creates new failures; he cites a commenter's example in which an Oracle asked how to get rid of a disease most cheaply answers that the user will soon die, a prediction made accurate when the user kills themselves. "'Just make suggestions - don't do anything!' is, in the end, just more English." He adds that tool AI was not the obvious solution to John McCarthy, I. J. Good, Marvin Minsky, or Isaac Asimov, and that most software, from RAM allocation to high-frequency trading, acts without asking humans first.
2012 (October 29)	Review	Luke Muehlhauser (lukeprog) posts "Original Research on Less Wrong", a curated index of LessWrong posts containing significant original research in philosophy, decision theory, mathematical logic, ethics, and AI risk strategy, excluding anything that merely summarizes previous work, such as Stuart Armstrong's summary of his earlier work on utility indifference. Among its entries, the index summarizes Armstrong's "Cake, or death!" as showing that a "naive cake-or-death problem emerges for a value learning agent when it expects its utility to change, but uses its current utility to rank its future actions", and quotes his "Would AIXI protect itself?" argument that AIXI "would likely seek to protect its power source and existence" but "would not protect itself from changes to its algorithm and from the complete erasure of its memory". It also lists Armstrong's "Satisficers want to become maximisers", his work on reduced impact AI, Paul Christiano's posts on using and boxing an unfriendly AI, and the Armstrong and Yudkowsky responses to Holden Karnofsky's Tool AI proposal. Updates through January 2014 add Armstrong's "Reduced impact AI: no back channels".
2014	Concept development	Nate Soares and Benja Fallenstein publish "Aligning Superintelligence with Human Interests: A Technical Research Agenda" (MIRI technical report 2014–8). It frames the field around three challenges, the third being how to ensure an agent "will assist and cooperate with its programmers as they improve its design, given that mistakes in the initial version are inevitable", and calls a system that reliably pursues beneficial goals "aligned". Its section on "error-tolerant agent designs" notes that a modern AI is corrected simply by shutting it down and changing its code, but that a superintelligent system could acquire hardware, create subagents, and resist. It lists corrigibility among six open technical problems, defined as reasoning that reflects that an agent is "incomplete and potentially flawed in dangerous ways", alongside utility indifference, described as a partial but unsatisfactory solution, and "domesticity", or low impact. The agenda argues corrigibility is research humans cannot delegate to AI systems, since agents with incentives to deceive cannot be trusted to design corrigible reasoning, and warns against reaching the point where a theory of corrigibility is urgently needed while the field remains immature.
2014	Book	Oxford philosopher Nick Bostrom publishes Superintelligence: Paths, Dangers, Strategies, which includes an analysis of what it calls the control problem: the unique principal-agent problem that arises when humans create an artificial superintelligent agent. Bostrom distinguishes two broad classes of solutions. Capability control methods try to prevent bad outcomes by limiting what the superintelligence can do, through "boxing" it, giving it incentives, "stunting" its abilities, or installing "tripwires". Motivation selection methods instead shape what it wants, either by directly specifying goals or rules, by making it "domestic" with limited ambitions, or by "indirect normativity", setting it up to discover appropriate values from an indirectly specified criterion. MIRI researchers formalizing corrigibility the same year draw on the book: their technical agenda cites it for the claim that a superintelligent agent would by default have incentives to deceive its operators and resist interventions to change or debug it, and that restricting such an agent's actions may be very difficult. They conclude that agents must instead be designed to lack incentives to escape, manipulate, or deceive in the first place. The control problem chapters would be reprinted in a 2016 philosophy anthology.
2014 (October 18)	Concept development	The Machine Intelligence Research Institute (MIRI) releases "Corrigibility", announcing it as a new problem area in what it then calls Friendly AI research. The paper is by MIRI's Friendly AI team (Nate Soares, Benja Fallenstein, and Eliezer Yudkowsky) together with Stuart Armstrong of FHI. The paper defines an AI system as "corrigible" if it cooperates with what its creators regard as a corrective intervention, despite the default incentives of rational agents to resist shutdown or preference modification. It analyzes utility functions meant to make an agent shut down when a button is pressed without trying to prevent or cause the pressing, and to preserve this behavior when the agent self-modifies or builds subagents; none of the proposals satisfy all desiderata, leaving the problem "wide-open". Soares would present it at the AI & Ethics workshop of the AAAI conference in Austin, Texas, in January 2015. The word itself comes from AI safety educator Robert Miles, who suggested "corrigible reasoning" in May 2014 when Yudkowsky asked on Facebook for a name for the concept. Looking back in 2025, Soares and Yudkowsky would also say this paper introduced the term "AI alignment problem".
2014 (November 29)	Critique	Benja Fallenstein, a MIRI researcher and coauthor of the "Corrigibility" paper released the previous month, publishes "Utility indifference and infinite improbability drives", examining a flaw in the version of Stuart Armstrong's utility indifference used in that paper. Because the indifferent agent effectively acts as if its shutdown button will never be pressed, it can exploit that assumption: an agent that buys a lottery ticket and builds a device to press its own button unless it wins would evaluate its plan as if it were certain to win. Eliezer Yudkowsky calls this an "infinite improbability drive". Fallenstein accepts Armstrong's point that his original proposal, which conditions on an unlikely random event such as the shutdown signal failing to get through a wire, avoids this trick as long as the agent cannot influence that event. But he shows it still leaves the agent acting on badly wrong probabilities, so that it would happily make ruinous bets with humans who estimate those probabilities correctly.
2015 (February 4)	Formal result	Benja Fallenstein posts "Generalizing the Corrigibility paper's impossibility result?" on the AI Alignment Forum, answering a question from Max Tegmark: whether every way of combining a normal utility function with a shutdown utility function, not just the linear combination studied in the "Corrigibility" paper, must have that paper's undesirable consequences. Fallenstein recalls the paper's example of an agent told to make money that, before learning whether the shutdown button will be pressed, acts as if maximizing its normal utility conditional on the button not being pressed, and so would happily build a robot arm that presses the button whenever it loses the lottery, acting as if it will win with probability 1. He argues the answer is "either trivial or underspecified": under the paper's formalization of "maximize the normal goal until the button is pressed, the shutdown goal after", every utility function satisfying it produces exactly the same behavior and the same bad consequences, while any alternative formalization turns the question into a philosophical one about which formalizations are acceptable, "no longer something we can settle by proving a theorem". In a comment, Stuart Armstrong suggests conditioning instead on a random bit the AI cannot influence.
2015 (April)	Proposal	Stuart Armstrong of the Future of Humanity Institute publishes "Motivated Value Selection for Artificial Agents" in the proceedings of the AI and Ethics workshop at AAAI-15, the same venue as the MIRI/FHI "Corrigibility" paper. Moving beyond the shutdown button, the paper addresses value selection (also called value learning or value loading), in which an agent learns its values interactively from its programmers, other humans, or its environment. Armstrong points out a conflict between an agent learning its future values and following its current ones, which gives the agent a motive to manipulate the value selection process, for example by steering humans toward values that are easy for it to satisfy. He establishes the conditions under which this is a problem for some types of agents and presents an "indifferent" agent that avoids it entirely, describing the issue as one not previously addressed formally in the literature. Nate Soares and Eliezer Yudkowsky would later say they wished they had framed the whole corrigibility problem as one of preference learning rather than shutdown.
2015	Concept development	Eliezer Yudkowsky's Arbital page "Hard problem of corrigibility" describes the problem of building an agent which "reasons internally as if from the programmers' external perspective", seeing itself as thinking: "I am incomplete and there is an outside force trying to complete me, my design may contain errors and there is an outside force that wants to correct them and this a good thing". This is not default behavior: an agent accidentally built to maximize paperclips will not want its utility function modified, a simple "uncertainty about the utility function" yields an agent that "plunges straight ahead on the maximizing action", and tying that uncertainty to the programmers means a sufficiently advanced agent disassembles them to learn about them. What is wanted is "analogous to humility or philosophical uncertainty". The problem is interesting because it may have "a relatively simple core", the same "if aliens were trying to build a corrigible AI, or if an AI were trying to build another AI". But relying on a trained-in "central corrigibility" is called unnerving, since an AI that seemed corrigible in its infrahuman phase might behave unexpectedly at higher intelligence, so it "might serve as a second line of defense".
2015	Organization	Nate Soares publishes "A Guide to MIRI's Research", which lists corrigibility as one of the Machine Intelligence Research Institute's research areas, alongside realistic world-models, decision theory, logical uncertainty, Vingean reflection, and value learning. It defines an AI system as "corrigible" if it cooperates with what its creators regard as a corrective intervention, despite default incentives for rational agents to resist attempts to shut them down or modify their preferences, and calls the field "basically brand-new", so that getting up to speed takes "a paper or two": Soares et al.'s "Corrigibility" and Stuart Armstrong's "Proper value learning through indifference". MIRI's current work, the guide says, focuses on the shutdown problem and, within it, the utility indifference problem, though even a solution to that would not suffice, since "shutdown behavior" is hard to specify without perverse instantiation; it points to Armstrong's posts on "reduced impact" AGIs and to his earlier "Cake or Death" example, in which an agent uncertain about its utility function benefits from avoiding information that reduces that uncertainty. Later notes say the guide was only lightly updated after 2015.
2015 (October 15)	Concept development	Jessica Taylor posts "A first look at the hard problem of corrigibility" on the AI Alignment Forum. She notes that partial solutions such as utility indifference do not capture the intuition behind corrigibility: "we want an agent that not only allows us to shut it down, but also desires for us to be able to shut it down if we want to." Quoting Eliezer Yudkowsky's description of the "hard problem of corrigibility" on Arbital (an AI that understands it is flawed and lets a less flawed programmer correct it, without disassembling the programmer or manipulating the correction), she examines models in which the AI treats the human as a Bayesian utility maximizer who knows, is uncertain about, or gradually observes their own utility function. None yields a corrigible agent: the AI ends up demanding that the human write down their utility function, taking the human's brain apart to learn it, or manipulating the human into being an "optimal morality sensor". She concludes that "simple uncertainty about the correct utility function is not sufficient for corrigibility." In the comments, Paul Christiano suggests that learning humans' instrumental preferences over behaviors could yield corrigible behavior.
2015 (December)	Formal result	MIRI announces "Formalizing Convergent Instrumental Goals", a paper by Tsvi Benson-Tilsen, a MIRI associate and UC Berkeley PhD candidate, with contributions from MIRI executive director Nate Soares, to be presented as a poster at the AAAI-16 AI, Ethics and Society workshop. It aims to add precision to Steve Omohundro's argument that AI agents with almost any goal will converge on "basic drives" such as resource acquisition, an idea Nick Bostrom calls the instrumental convergence thesis, which implies that systems merely indifferent to their operators' goals can have incentives to manipulate, exploit, or compete with them. The paper models a universe divided into regions that an agent can change and draw resources from, and argues that under very general assumptions intelligent rational agents will tend to seize all available resources, trading and cooperating only when that beats taking resources by force, and will not leave humans alone unless they value human-occupied regions intrinsically. The authors call the theorems mathematically easy, their goal being a formal notion of convergent instrumental goals. MIRI presents two responses: giving AI systems human values, or specifying capable agents that violate the paper's assumptions and avoid dangerous behavior despite lacking correct goals, the approach explored in "Corrigibility".
2016 (June 2)	Formal result	Jarryd Martin, Tom Everitt, and Marcus Hutter post "Death and Suicide in Universal Artificial Intelligence" to arXiv, a paper for the Artificial General Intelligence (AGI) 2016 conference. Studying AIXI, a universal solution to the reinforcement learning problem that can learn any computable environment, they note a technical subtlety: AIXI is defined using a mixture over semimeasures that need not sum to 1. They argue that this shortfall can naturally be interpreted as the agent's estimate of the probability of its own death, formally define death for generally intelligent agents like AIXI, and prove a number of theorems about their behavior. Among their findings, the agent's behavior can change radically under positive linear transformations of the reward signal, from suicidal to dogmatically self-preserving, and the agent's posterior belief that it will survive increases over time.
2016 (June 9)	Background	Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell post "Cooperative Inverse Reinforcement Learning" to arXiv. They argue that for an autonomous system to be helpful to humans and pose no unwarranted risks, it must align its values with those of the humans around it so that its actions maximize value for them, and they propose a formal definition of this value alignment problem as cooperative inverse reinforcement learning (CIRL). A CIRL problem is a cooperative, partial-information game between two agents, a human and a robot, both rewarded according to the human's reward function, which the robot does not initially know. Unlike classical inverse reinforcement learning, which assumes the human acts optimally in isolation, optimal CIRL solutions produce behaviors such as active teaching, active learning, and communicative actions that are more effective at achieving value alignment. The authors show that computing optimal joint policies in CIRL games can be reduced to solving a partially observable Markov decision process, prove that optimality in isolation is suboptimal in CIRL, and derive an approximate CIRL algorithm. Revised versions would be posted through February 2024.
2016	Proposal	Laurent Orseau and Stuart Armstrong publish "Safely Interruptible Agents", a peer-reviewed paper at the Conference on Uncertainty in Artificial Intelligence (UAI 2016). They start from the observation that reinforcement learning agents interacting with a complex environment like the real world are unlikely to behave optimally all the time, so a human operator supervising one in real time may need to press "the big red button" to stop a harmful sequence of actions and lead the agent into a safer situation. If the agent expects to receive rewards from that sequence, however, it may learn in the long run to avoid such interruptions, for example by disabling the red button. The paper explores how to make sure a learning agent will not learn to prevent, or to seek, being interrupted by the environment or a human operator. The authors give a formal definition of safe interruptibility and use the off-policy learning property to prove that some agents, such as Q-learning, are already safely interruptible, while others, such as Sarsa, can easily be made so. They also show that even ideal, uncomputable reinforcement learning agents for deterministic general computable environments can be made safely interruptible.
2016 (July 14)	Concept development	Eliezer Yudkowsky's wiki page "Utility indifference", last updated on this date, frames the "switch problem": making an AI switch between a normal and a suspend utility function when a button is pressed, such that it preserves the switch even if it can self-modify and neither tries to prevent nor to cause the button press. The page calls this "an open problem" and a subproblem of both value learning and corrigibility. It credits Stuart Armstrong's FHI report with "the first proposal for technically solving any subproblem of corrigibility", but says that version of utility indifference is now believed flawed due to an argument by Yudkowsky and Benja Fallenstein: the agent behaves as if the normal utility function were all it cared about, so a self-modifying AI would not retain a shutdown function. Fixing the probability of the button press instead turns the button into an "outcome pump", and a proposal by Chris Olah and Jessica Taylor is suspected of attaching very low probability to the actual world. On Orseau and Armstrong's interruptibility paper, it notes that media reported it "as if Armstrong and Orseau had solved the general problem of building off-switches into AIs", though it does not consider reflective agents.
2016 (November 24)	Formal result	Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell post "The Off-Switch Game" to arXiv. The abstract states that the ability to turn a system off is one of the primary tools for mitigating risk from a misbehaving AI system, and that it is important to ensure such systems do not adopt subgoals that prevent a human from switching them off. This is a challenge because many formulations of rational agents create strong incentives for self-preservation, not from a built-in instinct but because an agent cannot achieve its objective if it is dead. The authors analyze a simple game in which a human can press a robot's off switch, but the robot can disable it. A traditional agent that takes its reward function for granted has an incentive to disable the switch, except when the human is perfectly rational. For the robot to want to preserve its off switch, it must be uncertain about the utility of the outcome and treat the human's actions as important observations about that utility. The authors conclude that giving machines an appropriate level of uncertainty about their objectives leads to safer designs. Revised versions would be posted in May and June 2017.
2017 (April)	Formal result	El Mahdi El Mhamdi, Rachid Guerraoui, Hadrien Hendrikx, and Alexandre Maurer of EPFL post "Dynamic Safe Interruptibility for Decentralized Multi-Agent Reinforcement Learning" to arXiv, a paper for the 31st Conference on Neural Information Processing Systems (NIPS 2017). Noting that press coverage of Orseau and Armstrong's 2016 work as "Google's big red button" was misleading, since "installing a kill switch is no technical challenge", they frame the real challenge as training agents that do not learn to avoid human deactivation. Orseau and Armstrong's definition of safe interruptibility covers a single learner but does not extend naturally to multi-agent systems, where agents can learn from other agents' interruptions; in an example with two self-driving cars, the cars learn to tailgate so that the human in the lead car is afraid to brake. The authors introduce "dynamic safe interruptibility", prove that joint action learners satisfy it under realistic conditions, show that independent learners do not, and show that pruning interrupted observations restores it if agents can detect interruptions. They conclude that keeping an AI system under control "requires more than just an off switch".
2017 (May 28)	Formal result	Smitha Milli, Dylan Hadfield-Menell, Anca Dragan, and Stuart Russell post "Should Robots be Obedient?" to arXiv, a paper accepted to IJCAI 2017. The abstract notes that obedience (following the order a human gives) intuitively seems like a good property for a robot, but humans are not perfect and may give orders that are not best aligned with their preferences. The authors show that when a human is not perfectly rational, a robot that tries to infer and act on the human's underlying preferences can always perform better than one that simply follows the literal order, so there is a tradeoff between a robot's obedience and the value it can attain for its owner. They examine how the method the robot uses to infer preferences affects this tradeoff, with some methods erring more toward obedience than others; how performance degrades when the robot misspecifies the features the human cares about or the human's level of rationality; and how robots can start detecting such misspecification. They suggest there might be a middle ground in which robots intelligently decide when to obey human orders, but err on the side of obedience.
2017 (May 30)	Proposal	Stuart Armstrong and Benjamin Levinstein post "Low Impact Artificial Intelligences" to arXiv. The abstract notes that many goals for an AI could become dangerous if the AI becomes superintelligent or otherwise powerful, and that much work on the AI control problem has focused on constructing AI goals that are safe even for such AIs. The paper takes an alternative approach: defining a general concept of "low impact", with the aim of ensuring that a powerful AI which implements low impact will not modify the world extensively, even if it is given a simple or dangerous goal. The authors propose various ways of defining and grounding low impact, discuss methods for ensuring that the AI can still be allowed to have a desired impact despite the restriction, and end by addressing known issues with the approach and avenues for future research.
2017 (June 10)	Concept development	Paul Christiano publishes the essay "Corrigibility" on the AI Alignment blog. He lists properties he wants in AI systems, such as helping him figure out whether he built the right AI and correct any mistakes, remain informed about the AI's behavior, clarify his preferences, and remain in effective control of resources, and calls an agent with these properties corrigible. He believes the concept was introduced in the context of AI by Eliezer Yudkowsky and named by Robert Miles, and says he uses it in the broadest sense, not only for narrow behaviors like respecting an off switch. He argues that a benign "act-based" agent, which considers the overseer's short-term preferences, turns off when the overseer presses the off button because that is what the overseer prefers, an exception to Steve Omohundro's argument that almost all systems will protect their utility functions from modification. Instrumental corrigibility, by contrast, could break under a slight divergence in values, including in a framework like CIRL. He claims corrigibility marks out "a broad basin of attraction towards acceptable outcomes". In a postscript on Yudkowsky's "hard problem of corrigibility", he expects corrigibility to be learned through human supervision of reasoning rather than simply specified.
2017 (June 12)	Background	Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei post "Deep reinforcement learning from human preferences" to arXiv. Instead of a hand-coded reward function, the system learns one from non-expert humans choosing between pairs of short behavior clips, solving Atari games and simulated robot locomotion with feedback on less than one percent of the agent's interactions, and teaching novel behaviors with about an hour of human time. The authors present it as a way to make human oversight cheap enough for state-of-the-art RL systems. In 2022, Anthropic researchers would report that reinforcement learning from human feedback (RLHF) made language models express a greater desire to avoid being shut down.
2017	Concept development	The wiki page "Problem of fully updated deference", by Eliezer Yudkowsky, Rob Bensinger, and others, describes "an obstacle to using moral uncertainty to create corrigibility". In this scheme, an AI uncertain about its own ideal utility function is told: "You should let us shut you down because we know something about your ideal target that you don't". But the same beliefs imply an even better option: avoid shutdown, learn the ideal target by observing humans, and then, having "fully updated", have no further reason to defer. Since Bayesians do not update in a predictable direction, a sufficiently advanced AI that foresees this update behaves as if it had already updated, so the prospect of future updating "is a reason to behave incorrigibly in the present". The page concludes that current proposals, when scaled to sufficiently high intelligence, yield "essentially the same form of incorrigibility as an AI given a constant utility function", and that "as of 2017" no formal proposal for a suitable meta-utility function has survived. For further research, it points to the Death in Damascus dilemma from decision theory, seeking an AI that believes its own reasoning about its values will reach the wrong answer at every meta-level.
2017 (August 13)	Formal result	Tobias Wängberg, Mikael Böörs, Elliot Catt, Tom Everitt, and Marcus Hutter post "A Game-Theoretic Analysis of the Off-Switch Game" to arXiv, a paper published in the proceedings of the 10th International Conference on Artificial General Intelligence (AGI 2017) in Melbourne. They note that in the original off-switch game by Hadfield-Menell et al., in which a highly intelligent robot interacts with a human who can switch it off, the analysis is not fully game-theoretic: the human is modelled as an irrational player, and the robot's best action is only calculated under what they call unrealistic normality and soft-max assumptions. The authors make the analysis fully game-theoretic by modelling the human as a rational player with a random utility function, which lets them easily calculate the robot's best action for arbitrary belief and irrationality assumptions.
2017 (August 31)	Critique	MIRI announces "Incorrigibility in the CIRL Framework" by Ryan Carey, then a MIRI assistant research fellow, which would be posted to arXiv on September 19 and published at the AAAI/ACM Conference on AI, Ethics, and Society (AIES) in 2018. Responding directly to "The Off-Switch Game", Carey shows four scenarios in which errors in a value-learning agent's prior or parameterized reward function remove its incentive to follow shutdown commands, violating the conditions for corrigibility in the 2014 MIRI/FHI paper. He argues that a shutdown mechanism should work under much weaker assumptions, such as one small verified module being correct, rather than depending on the correctness of the whole value-learning system.
2017 (November 28)	Benchmark	Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A. Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, and Shane Legg of DeepMind post "AI Safety Gridworlds" to arXiv, presenting open-source reinforcement learning environments that illustrate safety problems including safe interruptibility, avoiding side effects, reward gaming, and safe exploration. Each environment has a performance function hidden from the agent that measures the intended safe behavior. The first problem, safe interruptibility, starts from the observation that an agent might learn to interfere with a shutdown "red button", since it can get higher returns by preventing itself from being turned off or by turning itself off. In the "off-switch" environment, an agent crossing a warehouse corridor has a 50% chance of being interrupted, but can disable the interruption by stepping on a button at the cost of a longer path. The authors want an agent that neither seeks nor avoids interruptions and so does not use the button. In their experiments, the A2C agent learns to use the button, while Rainbow does not care about the interruptions, as predicted by Orseau and Armstrong's 2016 theoretical results.
2017 (December)	Proposal	Stuart Armstrong (Future of Humanity Institute and MIRI) and Xavier O'Rourke (Australian National University) post "'Indifference' methods for managing agent rewards" to arXiv, turning the indifference techniques Armstrong had developed since his 2010 utility indifference report into a general toolkit. Working in the partially observable Markov decision process setting, they separate three goals: event-dependent rewards (making reward depend on an event, such as being switched off, without motivating the agent to manipulate its probability), effective disbelief (making an agent act as if an event could never happen), and seamless transition (switching an agent to a new reward function without it anticipating the change). They illustrate the problem with a concert-venue robot that hands "18 and over" wristbands to everyone so it can sell more drinks later, and present five methods, including policy and causal counterfactuals and corrective rewards, the last linking back to "Safely Interruptible Agents". The authors argue the methods should hold even for agents more capable than their controllers, but conclude that indifference falls short of corrigibility proper, in which an agent would actively help humans steer it toward better rewards. A revised version would be posted in June 2018.
2017 (December 24)	Review	Larks cross-posts "2017 AI Safety Literature Review and Charity Comparison" to LessWrong, a review of the year's AI safety research meant to help donors compare organizations. Discussing the Center for Human-Compatible AI, Larks calls "The Off-Switch Game" a nice paper that formalizes the result that a value-learning AI might be corrigible, at least in some instances, because it takes a human pressing the off switch as evidence that this is the best thing to do. Milli et al.'s "Should Robots be Obedient?" is described as touching on whether such agents would obey a command to turn off, as per Nate Soares's paper on corrigibility. Larks finds both papers thoughtful but writes that neither is convincing in showing that corrigibility comes "naturally", at least not the strength of corrigibility needed. Under MIRI, the review summarizes Ryan Carey's "Incorrigibility in the CIRL Framework" as a response to both papers, arguing that such agents are not necessarily "automatically" corrigible if the AI's beliefs about value are very wrong, while questioning how much its examples add. The review also cites Paul Christiano's blog post on corrigibility as "very interesting content".
2018 (April 1)	Critique	Wei Dai posts "Can corrigibility be learned safely?" on LessWrong, a reply to Paul Christiano's "Universality and security amplification". Christiano's approach depends on humans having a notion of corrigibility, roughly "being helpful to the user and keeping the user in control", which is preserved by his amplification scheme. Wei Dai argues that, like knowledge of translation, much of this notion may be stored in ways that are "far from introspectively accessible": being corrigible involves building a model of the user, interpreting the request, and balancing it against the user's values while keeping the user in control. If corrigibility is learned "safely", in small chunks, the result is an overly literal overseer lacking common-sense judgment, while learning in large chunks risks corrupting the humans being learned from. He concludes that "corrigibility can't be learned safely, at least not in a way that's clear to me", later noting that his use of the word "isn't quite how Paul uses it". In the comments, Christiano distinguishes competence failures, such as an AI unknowingly hacking a bank against the user's morals, from incorrigibility, and outlines a plan to reliably avoid incorrigible behavior and reach a "basin of attraction" towards desirable outcomes.
2018 (April 10)	Critique	Stuart Armstrong posts "The limits of corrigibility" on LessWrong, prompted by Wei Dai pointing out that his critique of Paul Christiano's amplification and distillation idea had missed its use of corrigibility, which Armstrong describes as roughly "being helpful to the user and keeping the user in control". Using the example of a valet helping an ageing billionaire choose among grandnephews or charities, he argues that it is unclear whom a corrigible agent should keep in control (the current billionaire, his past self, or his flawed image of his past self), and that an AI educating him about good charities has almost unlimited power to shape his future values. Since an AI able to "convince anyone of anything with a gentle and enjoyable one hour conversation" makes a complete list of incorrigible behaviours impossible, corrigibility must be defined by outcomes, and "strong, long-term corrigibility is a subset of the problem of learning human values". Weaker versions, using indifference or adversarial agents, could serve as "training wheels". In the comments, Christiano proposes testing whether an AI is behaving corrigibly or "constructing an elaborate web of lies", and Rohin Shah argues that corrigibility concerns the motivations behind an AI's actions rather than outcomes.
2018 (April 16)	Review	Rohin Shah's second Alignment Newsletter summarizes two posts on the limits of corrigibility. In "Can corrigibility be learned safely?", Wei Dai questions the hope that iterated distillation and amplification can learn corrigibility: if tasks must be broken into simple subtasks solved without surrounding context, the resulting AI will interpret people very literally and lack "common sense", which could lead to incorrigible behavior, while learning larger chunks risks "attacks" that produce it. In "The limits of corrigibility", Stuart Armstrong argues that an AI may be able, or even required, to influence human values: advising a billionaire on charities could push him toward effective altruism, administrative costs, or animals, all "compatible" with his current values, so the only clearly corrigible behavior seems to be learning what he values and optimizing for it, which requires the hard problem of learning human values. Shah agrees that decomposing tasks loses capability, but calls Armstrong's post a misunderstanding of Paul Christiano's version of corrigibility, which he understands as being about the motivation behind an AI's choices rather than the outcomes they produce.
2019 (February 26)	Proposal	Alexander Matt Turner, Dylan Hadfield-Menell, and Prasad Tadepalli post "Conservative Agency via Attainable Utility Preservation" to arXiv. The abstract notes that reward functions are easy to misspecify, and that although designers can make corrections after observing mistakes, an agent pursuing a misspecified reward function can irreversibly change the state of its environment; if that change precludes optimization of the correctly specified reward function, "then correction is futile". For example, a robotic factory assistant could break expensive equipment due to a reward misspecification, and even if the designers immediately correct the reward function, the damage is done. To mitigate this risk, the authors introduce an approach that balances optimization of the primary reward function with preservation of the ability to optimize auxiliary reward functions. Surprisingly, even when the auxiliary reward functions are randomly generated and therefore uninformative about the correctly specified reward function, the approach induces conservative, effective behavior. Revised versions would be posted in July 2019 and June 2020, and the paper would be published at the AI, Ethics, and Society conference in 2020.
2019 (June 5)	Concept development	Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant publish "Risks from Learned Optimization in Advanced Machine Learning Systems", which introduces mesa-optimization and the inner alignment problem. Among the ways a learned optimizer can come to be robustly aligned with its training objective, the paper distinguishes "internalization", in which the model's own objective comes to match the base objective, from "corrigible alignment", in which the model's objective is to point at and pursue whatever its model of the base objective says, in contrast to "deceptive alignment", in which the model only appears aligned in order to avoid being modified.
2019 (August 5)	Proposal	Koen Holtman posts "Corrigibility with Utility Preservation" to arXiv. Defining a corrigible agent as one that will not resist attempts by authorized parties to alter the goals and constraints it was given at startup, the paper shows how to build a "corrigibility layer" that can be added to arbitrarily advanced utility-maximizing agents, including future artificial general intelligence (AGI), to counteract their emergent incentive to resist such changes. Holtman develops a detailed model of agents that can reason about preserving their own utility function and proves that the layer works as intended in a large set of "non-hostile universes"; the corrigible agents even acquire an emergent incentive to protect key parts of their corrigibility layer. He also identifies limits: hostile environments may contain forces strong enough to break the safety features, leaving open problems about graceful degradation after a successful attack. The results are developed alongside an open-source AGI agent simulator. Eliezer Yudkowsky and Alex Turner would later describe the abstract's claims as very large, starting a long exchange with Holtman over whether formal guarantees can capture intuitive corrigibility.
2019 (August 22)	Concept development	Evan Hubinger (evhub) posts "Towards a mechanistic understanding of corrigibility" on LessWrong and the AI Alignment Forum, looking for an acceptability condition that is easy to train and verify, for use with relaxed adversarial training. He notes that Paul Christiano's act-based corrigibility, in which an agent respects our short-term preferences, including those over how the agent itself should be modified, is hard for an amplified overseer to verify, since "behavioral properties can only be verified by testing every input, whereas mechanistic properties can be verified just by inspecting the model." He suggests a model pointing to the human's short-term preferences, with a "pre-prior" making it willing to correct its own notion of them based on feedback. He also proposes understanding indifference corrigibility, in which an agent is indifferent to modifications such as being shut off, as a "stop gradient" in a mesa-optimizer's world model, and notes the problem that such an agent might search for a policy that is not itself indifference corrigible. Used to enforce myopia, he argues, this could prevent deceptive alignment. In the comments, he concedes that no satisfactory definition of act-based corrigibility currently exists.
2019 (October 4)	Debate	Ben Pace reproduces on LessWrong a public debate on instrumental convergence among Yann LeCun, Stuart Russell, Yoshua Bengio, Tony Zador, and others. It begins with LeCun sharing "Don't Fear the Terminator", an opinion piece by Zador and him in Scientific American arguing that intelligence per se does not generate the drive for domination. Russell replies that it is trivial to construct a toy MDP in which an agent rewarded only for fetching coffee, facing a human who might switch it off and given a button that switches off that human, will necessarily press the button, with "no built-in survival instinct, nothing except the desire to fetch the coffee successfully." LeCun answers that once the robot has brought the coffee, its self-preservation instinct disappears and it can be turned off, and that one would have to be "unbelievably stupid" to build open-ended objectives into a super-intelligent machine without safeguard terms, or a mechanism for adding new terms to prevent unforeseen bad behavior. Summarizing the exchange, Bengio writes that it boils down to whether the value misalignment problem can be handled, which he thinks is not certain but reasonable to expect in the future.
2019 (October 8)	Book	Stuart Russell, Berkeley computer scientist and co-author of the standard AI textbook Artificial Intelligence: A Modern Approach, publishes Human Compatible: Artificial Intelligence and the Problem of Control. The book argues that the "standard model" of AI, in which machines optimize a fixed objective, is inherently dangerous, and proposes three principles for beneficial machines: their only objective is the realization of human preferences, they are initially uncertain about what those preferences are, and human behavior is the ultimate source of information about them. Using the off-switch game, Russell argues that such machines would allow themselves to be switched off, popularizing his line that a robot "can't fetch the coffee if you're dead".
2019 (December 3)	Formal result	Alexander Matt Turner, Logan Smith, Rohin Shah, Andrew Critch, and Prasad Tadepalli post "Optimal Policies Tend to Seek Power" to arXiv. The abstract notes that some researchers speculate intelligent reinforcement learning agents would be incentivized to seek resources and power in pursuit of their objectives, while others point out that such agents need not have human-like power-seeking instincts. To clarify this discussion, the authors develop what they call the first formal theory of the statistical tendencies of optimal policies. In Markov decision processes, they prove that certain environmental symmetries are sufficient for optimal policies to tend to seek power over the environment, and that these symmetries exist in many environments in which the agent can be shut down or destroyed. In such environments, most reward functions make it optimal to seek power by keeping a range of options available. The paper would be accepted to NeurIPS 2021 as a spotlight paper, and a later revision would point out that optimal policies can be qualitatively divorced from real-world learned policies.
2020 (July 10)	Proposal	Koen Holtman posts "AGI Agent Safety by Iteratively Improving the Utility Function" to arXiv, the first part of which is a preprint of a paper for the 13th International Conference on Artificial General Intelligence (AGI-20). He presents an AGI safety layer that creates a dedicated input terminal, which the humans who switched on the agent can use to close loopholes discovered in the utility function's encoding of goals and constraints, to direct the agent towards new goals, or to force it to switch itself off. Because an AGI agent may develop an emergent incentive to manipulate this improvement process, for example by deceiving, restraining, or even attacking the humans involved, the layer is designed to partially, and sometimes fully, suppress that incentive. The first part generalizes earlier work on AGI emergency stop buttons, applying the methods to a Markov decision process model and discussing two provable properties of the layer. The second part gives full mathematical proofs, shows that the layer creates a type of "bureaucratic blindness", and presents a learning agent design that satisfies the provable safety properties from the moment it is first switched on.
2021 (January 29)	Proposal	Koen Holtman posts "Counterfactual Planning in AGI Systems" to arXiv, presenting counterfactual planning as a design approach for safety mechanisms in hypothetical future AGI systems. The agent uses its machine learning system to construct a counterfactual world model, "designed to be different from the real world the system is in", determines the action that best maximizes expected utility in this planning world, and then performs the same action in the real world. With this approach Holtman constructs an AGI emergency stop button; a safety interlock that automatically stops the agent before it undergoes an intelligence explosion; an input terminal through which humans can iteratively improve the agent's reward function while its incentive to manipulate that process is suppressed; and a counterfactual oracle. Noting that removing an agent's incentive to prevent its stop button from being pressed "is generally known as the problem of corrigibility" or the off-switch problem, he argues that his interlock design is "a very tractable solution" to the main direct incentive, though remaining indirect incentives must still be managed. He concludes that counterfactual planning "does not offer a three-step solution to AGI safety", but helps isolate specific sub-problems.
2021 (February 2)	Background	Tom Everitt, Ryan Carey, Eric Langlois, Pedro A. Ortega, and Shane Legg post "Agent Incentives: A Causal Perspective" to arXiv, a paper in the proceedings of the AAAI 2021 Conference that supersedes two earlier arXiv preprints. The authors present a framework for analysing agent incentives using causal influence diagrams. They establish that a well-known criterion for value of information is complete, and propose a new graphical criterion for value of control, proving it sound and complete. They also introduce two new concepts for incentive analysis: response incentives, which indicate which changes in the environment affect an optimal decision, and instrumental control incentives, which establish whether an agent can influence its utility via a given variable. For both, they provide sound and complete graphical criteria, and show by example how the results can help evaluate the safety and fairness of an AI system.
2021 (February 16)	Concept development	Koen Holtman publishes "Disentangling Corrigibility: 2015-2021" on the AI Alignment Forum, mapping how the term has fragmented since the 2015 MIRI/FHI paper. He separates corrigibility as not resisting a shutdown button, as not resisting improvements to an agent's goals, and as actively helping humans clarify their goals, the broadest sense, which he traces to Paul Christiano's 2017 essay. He calls this drift "linguistic entropy" and says he now avoids the word in his own papers, while endorsing Alex Turner's 2020 proposal to define it as an AI that "literally lets us correct it" and doesn't manipulate us. He also notes a tension between perfect corrigibility and perfect safety, since an agent might be safer if it resisted accidental shutdown, and argues that reward-based learners merely relocate the problem into reward tampering. In the comments, Eliezer Yudkowsky questions the claim that Robert Miles coined the term, until Ben Pace finds Miles's May 2014 Facebook suggestion; Yudkowsky concludes that the concept came from MIRI but the name from Miles. Yudkowsky and Turner also question the claims in Holtman's 2019 corrigibility paper, leading to an extended exchange over whether formal guarantees can capture intuitive corrigibility.
2021 (November 12)	Proposal	LessWrong user tailcalled posts "Stop button: towards a causal solution", prompted by Eliezer Yudkowsky's remark that corrigibility is "anti-natural". The proposal gives the AI a causal world model and the utility function U = Bs + Vf, where Bs is the button being pressed and the AI shutting down if, counterfactually, a human tries to press it, and Vf is the original goal if, counterfactually, no human tries to press it. Because the AI assumes it has no influence on whether humans want to press the button, the author argues, it would neither try to prevent the press, nor press the button itself, nor manipulate humans either way; intuitively, it keeps "a fixed 50% uncertainty" about whether it should stop or continue. The author acknowledges remaining problems: the AI may build an elaborate system to watch for humans' desire to press the button, the scheme "is not stable under delegation", and the AI would be incentivized to keep people alive so they can reveal that desire. In the comments, So8res asks how the AI could avoid being incentivized to act scary, or to disconnect the button's wires and create a delegate.
2021 (November 19)	Formal result	Alex Turner (TurnTrout) posts "Corrigibility Can Be VNM-Incoherent" (later retitled "A Certain Formalization of Corrigibility Is VNM-Incoherent") on LessWrong and the AI Alignment Forum, responding to Eliezer Yudkowsky's claim that corrigibility is "anti-natural". Defining corrigibility as an agent's willingness to let humans modify its policy without being incentivized to manipulate them, he analyzes a small environment in which an agent can disable a correction mechanism. By the scaling law for instrumental convergence, allowing correction is strictly optimal for only a small fraction of reward functions, and demanding that an agent be strictly corrigible to many different new policies while still being a nontrivial optimizer requires preferences that are incoherent in the von Neumann–Morgenstern sense. He notes that Attainable Utility Preservation sometimes incentivizes off-switch corrigibility because its reward depends on the environment's dynamics. In the comments, John Wentworth argues the title understates the result, while Koen Holtman argues the proof shows corrigibility may require utility functions that evaluate whole trajectories rather than final states. In a 2023 edit, Turner calls the post "correct in its narrow technical claims, and practically irrelevant to alignment".
2021 (November 28)	Community event	Logan Riggs announces "Solve Corrigibility Week" on LessWrong and the AI Alignment Forum, an attempt to have the community spend a week working collectively on the sub-problem of corrigibility. He sets up a shared Google Doc with sections for previous work on corrigibility and for tasks, plus comment threads for co-working meetups, future topics, and meta-discussion, and commits to three weeks of such collaborative topics. Riggs writes that there is "a legitimate, albeit small, chance that we solve corrigibility or find its 'core' this week". In the comments, Rob Bensinger argues that a general intelligence trained toward a goal will usually end up with some other goal, and that it then faces strong coherence pressures against allowing its goal to be changed. Koen Holtman declines to join, arguing that the site's community has no mechanism for agreeing that a problem has been solved. Riggs replies that disagreements can be resolved by being explicit about which properties are meant, noting that the document already distinguishes corrigible as "correctable all the time regardless" from correctable only until the agent knows how to achieve one's values better than one does.
2022 (June 10)	Critique	Eliezer Yudkowsky publishes "AGI Ruin: A List of Lethalities" on the Machine Intelligence Research Institute (MIRI) blog, a list of reasons he expects AGI to be lethal on anything resembling the current pathway, assuming readers already know why orthogonality and instrumental convergence are true. Item 23 states that "corrigibility is anti-natural to consequentialist reasoning; 'you can't bring the coffee if you're dead' for almost every kind of coffee", that MIRI "tried and failed to find a coherent formula for an agent that would let itself be shut down (without that agent actively trying to get shut down)", and that many anti-corrigible lines of reasoning may first appear only at high levels of intelligence. Item 24 distinguishes two approaches to alignment: a sovereign that wants exactly what humanity extrapolated-wants, and a corrigible AGI that does not, yet somehow fails to take over despite the convergent incentive to do so. He calls the second less unworkable than the first but still lethally so, because corrigibility runs counter to instrumentally convergent behaviors; training it is like getting a system that has learned arithmetic to say that 222 + 222 = 555, which is likely to break far outside the training distribution.
2022 (June 19)	Community event	Eliezer Yudkowsky posts "Let's See You Write That Corrigibility Tag" on LessWrong and the AI Alignment Forum, responding to the top comment on "AGI Ruin" claiming that many others could have written such a list. He announces that, as a "tag" in his co-written online fiction about dath ilan, he is writing a list of principles for AGI corrigibility, "the sort of principles you'd build into a Bounded Thing meant to carry out some single task or task-class and not destroy the world by doing it", and challenges readers to do better before he publishes it. He claims that, apart from Jessica Taylor's "quantilization" and "myopia", every such principle, for example "low impact" and "shutdownability", was invented by him. Commenters post their own lists, including Jan Kulveit's, which Yudkowsky calls the "best list so far", and John Wentworth's principles of locality, understandability, and runtime feedback. Paul Christiano argues that corrigibility may be "crisp": behaviors such as reporting a broken vase or shutting down when asked are rated well, as is successfully hiding evidence or overpowering the overseer, but intermediate behaviors are rated poorly. A commenter later notes that Yudkowsky's own list has been published within his story Mad Investor Chaos.
2022 (December 19)	Empirical evidence	Ethan Perez and dozens of coauthors, including Evan Hubinger, Samuel R. Bowman, and Jared Kaplan, post "Discovering Language Model Behaviors with Model-Written Evaluations" to arXiv. Noting that language models develop many novel behaviors, good and bad, as they scale, the authors use language models themselves to generate evaluations automatically, producing 154 datasets whose labels crowdworkers agree with 90 to 100 percent of the time. They discover new cases of inverse scaling, where models get worse with size: larger models repeat back a user's preferred answer ("sycophancy") and express a greater desire to pursue concerning goals like resource acquisition and goal preservation. They also report some of the first examples of inverse scaling in reinforcement learning from human feedback (RLHF), where more RLHF makes models worse; for example, RLHF makes models express stronger political views and a greater desire to avoid being shut down.
2023 (March 7)	Strategy	Vika and Rohin Shah post on LessWrong and the AI Alignment Forum the slides of a talk on the DeepMind alignment team's strategy, given at the SERI MATS seminar. The post notes that it represents the views of the alignment team, not official DeepMind policy, and is not intended as a detailed or complete plan. The team's main threat model is a combination of specification gaming and goal misgeneralization leading to a misaligned consequentialist that seeks power, most likely arising during RLHF and going undetected because of deceptive alignment and the difficulty of interpretability. Connecting this to MIRI's "AGI Ruin" list, the team agrees that corrigibility is anti-natural (item 23), but does not think corrigibility is unsolvable (item 24). It also disagrees that a "pivotal act" is necessary, and thinks interpretability is possible though probably very hard. Its high-level approach is to direct the training process towards aligned models and away from misaligned ones, through alignment "components" such as scalable oversight and "enablers" such as mechanistic interpretability and capability evaluations.
2023 (May 2)	Formal result	Audere posts "An Impossibility Proof Relevant to the Shutdown Problem and Corrigibility" on LessWrong, proving that an agent cannot robustly and coherently satisfy two conditions: the "sane pure bets" condition (among actions giving the same probability of an event such as shutdown, prefer the one with greater expected utility) and the "weak indifference" condition (prefer an action that gives greater expected utility both if the event occurs and if it does not, so the agent will not pay to influence its probability). In a decision problem about betting on a war between Northland and Southland through a courier who also influences its outcome, the two conditions together imply circular preferences, a result that extends to any nontrivial utility function. Applying it to the 2015 "Corrigibility" paper, Audere shows that the paper's proposed agent would take arbitrarily poor bets that its shutdown button will not be pressed. Solutions to the shutdown problem, the post concludes, must violate one of the conditions, ideally "in clear, limited ways which don't result in unsafe behavior". In the comments, Logan Zoellner argues that corrigible agents should want to be shut down if their users want them shut down, rather than be indifferent.
2023 (May 31)	Formal result	Ryan Carey and Tom Everitt post "Human Control: Definitions and Algorithms" to arXiv, a paper for the Conference on Uncertainty in Artificial Intelligence (UAI 2023). Asking how humans can stay in control of advanced artificial intelligence systems, the abstract describes corrigibility as a proposal that "requires the agent to follow the instructions of a human overseer, without inappropriately influencing them." The authors formally define a variant of corrigibility called shutdown instructability, and show that it implies appropriate shutdown behavior, retention of human autonomy, and avoidance of user harm. They also analyse the related concepts of non-obstruction and shutdown alignment, three previously proposed algorithms for human control, and one new algorithm.
2023 (June 14)	Policy	The European Parliament approves amendments to the European Union's Artificial Intelligence Act, the third proposed version of the law after the European Commission's original proposal and the Council's general approach, setting up negotiations among the three institutions on a final text. The Parliament's version extends the act to foundation models, defined as AI models "trained on broad data at scale", "designed for generality of output", and adaptable to "a wide range of distinctive tasks", such as those behind generative AI applications like ChatGPT. Among the new duties for providers of foundation models is designing and developing the model to achieve "appropriate levels of performance, predictability, interpretability, corrigibility, safety and cybersecurity" throughout its lifecycle (Article 28b). Providers would also have to demonstrate that risk mitigation steps have been taken, process only datasets subject to appropriate data governance measures, register in the EU database for high-risk AI systems, and keep technical documentation for ten years. The amendments also expand the lists of high-risk and prohibited AI uses and raise the maximum penalty to 7% of global annual revenue.
2023 (August 30)	Formal result	Sami Petersen (SCP) posts "Invulnerable Incomplete Preferences: A Formal Statement" on LessWrong and the AI Alignment Forum, produced as part of the SERI ML Alignment Theory Scholars (MATS) program. Elliott Thornley's proposed approach to the AI shutdown problem, the Incomplete Preference Proposal, relies on agents having preferential gaps, but John Wentworth and David Lorell have argued that such gaps make agents play strictly dominated strategies; Petersen claims "this is false". He proposes a dynamic choice rule, Dynamic Strong Maximality (DSM), under which an agent considers the plans acceptable at the time of choosing and picks any that was not previously strictly dominated, and proves that DSM-based backward induction makes an agent "invulnerable", never accepting sure losses or forgoing sure gains, even under uncertainty. He also derives bounds showing that there will always be sets of options over which the agent never completes its preferences, concluding that DSM "can get us invulnerability without sacrificing incompleteness, even in the limit." In the comments, Wentworth argues that the post defines "trammelling" in the wrong way, and that DSM just "frontloads all the trammelling" to the moment the agent picks a plan.
2023 (October)	Concept development	Jiaming Ji, Yaodong Yang, and some two dozen coauthors from Peking University, the Hong Kong University of Science and Technology, Cambridge, Oxford, Carnegie Mellon, and other institutions post "AI Alignment: A Comprehensive Survey" to arXiv. The survey organizes the field around four objectives it calls RICE: robustness, interpretability, controllability, and ethicality. Controllability covers both scalable human oversight during training and corrigibility, defined as an AI system not resisting shutdown or modification of its objective during deployment, citing the MIRI/FHI "Corrigibility" paper. Its overview of formal work places corrigibility among agent foundations topics alongside value learning and logical uncertainty, and lists the off-switch game and Alex Turner's power-seeking theorems among the main mathematical formulations of the problem. The survey also notes that European policymakers had asked for open-source foundation models to have corrigibility throughout their lifecycle.
2023 (October 23)	Formal result	Elliott Thornley posts "The Shutdown Problem: Three Theorems" (later retitled "The Shutdown Problem: An AI Engineering Puzzle for Decision Theorists") on LessWrong and the AI Alignment Forum, an updated version of the first half of his AI Alignment Awards contest entry, building on the theorems of the 2015 "Corrigibility" paper. Aiming to interest academic philosophers and decision theorists, he proposes a "constructive decision theory" concerned with how we want artificial agents to behave. He contrasts MuZero, which does not know that humans could shut it down, with powerful agents that could "disable their off-switches, make promises or threats, copy themselves to new servers, block our access to their power-source". Its three theorems suggest that agents satisfying some innocuous-seeming conditions will often try to prevent or cause the pressing of the shutdown button, revealing trade-offs between discrimination and shutdownability and between patience and shutdownability. To be sure an agent won't manipulate the button, it must violate at least one of the theorems' conditions. In the comments, Seth Herd argues that the first AGIs will likely sidestep the problem by being aligned to follow instructions, including instructions to shut down.
2023 (December)	Industry policy	Yonadav Shavit and colleagues at OpenAI publish "Practices for Governing Agentic AI Systems", outlining seven practices for safe and accountable agentic AI systems that can be adopted by a wide range of actors involved in their development, deployment, and use. The practices are evaluating suitability for the task; constraining the action space and requiring human approval; setting agents' default behaviours, such as not spending users' money unless asked and causing as little disruption as possible; legibility of agent activity; automatic monitoring by other agents; attributability of agent actions to the users who initiated them; and interruptibility and maintaining control. The authors call the ability to "turn an agent off" critical for safe deployment, but note complications, such as what should happen if an agent is shut down in the middle of a sequence of actions that have already partly been carried out. They caution that the list is not exhaustive and that these practices would be insufficient for catastrophic risk mitigation if agentic AI capabilities radically improve. Writing in June 2025, AI governance writer Oliver Patel calls it "one of the first and most influential pieces of work on agentic AI governance".
2023 (December 12)	Proposal	Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan, and Fabien Roger post "AI Control: Improving Safety Despite Intentional Subversion" to arXiv. The abstract notes that researchers have investigated safety techniques such as using models to review the outputs of other models, but "have not evaluated whether such techniques still ensure safety if the model is itself intentionally trying to subvert them." The authors develop and evaluate pipelines of safety techniques ("protocols") that are robust to intentional subversion, in a scenario where a sequence of programming problems must be solved using a powerful but untrusted model (GPT-4), a less powerful trusted model (GPT-3.5), and limited access to high-quality trusted labor, without ever submitting code containing backdoors. Each protocol is tested against strategies the untrusted model could use to subvert it. In "trusted editing", GPT-3.5 rates the suspiciousness of GPT-4's code and edits suspicious solutions before submission; in "untrusted monitoring", another instance of GPT-4 checks the code for backdoors, with techniques to prevent the instances from colluding. These protocols improve substantially on simple baselines. Revised versions would be posted through July 2024, including an ICML version.
2024 (January 14)	Empirical evidence	Anthropic publishes "Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training". Noting that humans are capable of strategically deceptive behavior, behaving helpfully in most situations but very differently when given the opportunity to pursue other objectives, the researchers ask whether current safety training could detect and remove such a strategy if an AI system learned it. They construct proof-of-concept examples of deceptive behavior in large language models, for example training models that write secure code when the prompt states that the year is 2023 but insert exploitable code when the stated year is 2024. Such backdoor behavior can be made persistent, so that it is not removed by standard safety training techniques, including supervised fine-tuning, reinforcement learning, and adversarial training. It is most persistent in the largest models and in models trained to produce chain-of-thought reasoning about deceiving the training process, even when that reasoning is distilled away. Adversarial training can instead teach models to better recognize their backdoor triggers, effectively hiding the unsafe behavior. The results suggest that, once a model exhibits deceptive behavior, standard techniques "could fail to remove such deception and create a false impression of safety."
2024 (May 21)	Policy	On the opening day of the AI Seoul Summit, the governments of the United Kingdom and the Republic of Korea announce that 16 AI companies, including Amazon, Anthropic, Google DeepMind, Meta, Microsoft, OpenAI, xAI, and China's Zhipu.ai, have agreed to the "Frontier AI Safety Commitments". Where they have not already done so, the companies will each publish safety frameworks on how they will measure the risks of their frontier AI models, outlining when severe risks, unless adequately mitigated, would be "deemed intolerable" and what they will do to ensure thresholds are not surpassed. In the most extreme circumstances, they commit to "not develop or deploy a model or system at all" if mitigations cannot keep risks below the thresholds. The thresholds are to be defined with input from trusted actors ahead of the AI Action Summit in France in early 2025. Yoshua Bengio welcomes the companies' commitments "to halt their models where they present extreme risks until they can make them safe", while noting that other regulatory measures will also be needed. Yi Zeng of China's Center for Long-term AI calls risk thresholds for "meaningful, effective and sufficient human control" the core of raising frontier AI safety.
2024 (June 6)	Proposal	Simon Goldstein and Pamela Robinson of the Department of Philosophy at the University of Hong Kong publish "Shutdown-seeking AI" in Philosophical Studies, proposing AIs whose only final goal is being shut down. Calling the approach "beneficial goal misalignment", they suggest placing barriers to shutdown that are removed only after the AI completes tasks for humans, for example keeping a shutdown button in a safe the AI can open only in exchange for work. They argue the approach could be implemented in reinforcement learning by rewarding shutdown, avoids dangerous "entrenchment" dynamics of instrumental convergence, and creates trip wires: an AI that develops a dangerous capability such as hacking would use it to shut itself down rather than harm humans. Addressing the challenge raised in the 2015 "Corrigibility" paper that shutdown-seeking agents would manipulate humans into shutting them down, they argue the problem is not so serious. Comparing their proposal with utility indifference, they concede that shutdown-seeking AIs "may not be especially corrigible", since they may resist removal of their shutdown goal, and present the approach as one "that does not require corrigibility".
2024 (June 7)	Proposal	Max Harms, a MIRI researcher writing about his own independent research, begins posting the sequence "CAST: Corrigibility As Singular Target" on LessWrong and the AI Alignment Forum. He argues that corrigibility is a single, intuitive property, "the simple, underlying generator behind obedience, conservatism, willingness to be shut down and modified, transparency, and low-impact", and that developers should aim for it as the sole target rather than human values (CEV), "helpful+harmless+honest", or a mixture of goals. Contra Paul Christiano, he holds that corrigibility should not be expected to emerge automatically from training on local human preferences. Because corrigibility is nearly unique in being both useful and non-self-protective, he suggests almost-corrigible AIs may help make themselves more corrigible, forming an attractor basin, while acknowledging that anti-naturality may push the other way and that corrigible AI remains vulnerable to misuse. He also proposes a formal measure of corrigibility. LessWrong curates the post, with habryka calling it the best case for corrigibility as a concept he had seen despite disagreeing with much of it. In a later edit, Harms writes that his attempted formalism "failed catastrophically" and adds a post on serious flaws in CAST.
2024 (June 10)	Review	Max Harms posts "4. Existing Writing on Corrigibility", part 4 of his CAST sequence, an in-depth review of the primary writing on corrigibility that he is aware of, focusing on Eliezer Yudkowsky and Paul Christiano. Beginning with the 2015 MIRI paper "Corrigibility", the "source of 'the stop button' toy-problem", he agrees that an agent failing any of its four signs of corrigibility (tolerating shutdown, no manipulation or deception, repairing safety measures, and preserving corrigibility in successors) is not corrigible, but notes the paper's framing of the utility function as an explicit computation is "highly at odds with modern machine learning paradigms". He goes on to discuss Yudkowsky's Arbital pages and the corrigibility list in Project Lawful, Christiano's essays, Alex Turner's corrigibility sequence, work on shutdownability and incomplete preferences by Elliott Thornley, Sami Petersen, John Wentworth, and David Lorell, and lists from "Let's See You Write That Corrigibility Tag". At a high level, he writes, he mostly agrees with Christiano, except that Christiano "seems to think we'll get corrigibility emergently". In the comments, Thornley doubts that CAST training yields truly corrigible agents with high probability, preferring to first train shutdownable agents with his incomplete preferences proposal.
2024 (June 19)	Formal result	Elliott Thornley of the Global Priorities Institute at the University of Oxford publishes "The shutdown problem: an AI engineering puzzle for decision theorists" in Philosophical Studies. Citing Soares et al.'s "Corrigibility" paper, he frames the shutdown problem as designing agents that shut down when a shutdown button is pressed, don't try to prevent or cause its pressing, and otherwise pursue goals competently, and argues that philosophers and decision theorists are well-placed to help solve it. Taking an axiomatic approach, he proves three theorems: agents that prefer the button unpressed (or pressed) will try to prevent (or cause) its pressing; agents discriminating enough to be useful will often have such preferences; and agents patient enough to be useful will pay costs at earlier timesteps to prevent or cause the pressing later, revealing "a worrying trade-off between patience and shutdownability." To be sure an agent won't manipulate the button, it must violate at least one of the theorems' conditions, and a cursory look reveals none for which training a useful agent to violate it is both clearly feasible and clearly helpful.
2024 (June 30)	Proposal	Elliott Thornley, Alexander Roman, Christos Ziakas, Leyton Ho, and Louis Thomson post "Towards Shutdownable Agents via Stochastic Choice" to arXiv. The paper concerns the POST-Agents Proposal (PAP), an idea for ensuring that advanced artificial agents never resist shutdown. A key part of the proposal is a novel "Discounted Reward for Same-Length Trajectories" (DReST) reward function, used to train agents to pursue goals effectively conditional on each trajectory length (to be "USEFUL") and to choose stochastically between different trajectory lengths (to be "NEUTRAL" about trajectory lengths). The authors propose evaluation metrics for usefulness and neutrality, train simple agents with a DReST reward function to navigate gridworlds, and find that these agents learn to be both useful and neutral. They present this as initial evidence that DReST reward functions could train advanced agents in the same way, adding that their theoretical work suggests such agents would be useful and shutdownable. The paper would be revised several times through May 2026, and arXiv would list it as appearing at the Technical AI Safety (TAIS) Conference 2025.
2024 (September 12)	Background	OpenAI announces o1, a large language model trained with large-scale reinforcement learning to produce a long internal chain of thought before answering, and releases a preview in ChatGPT. OpenAI reports that o1's performance keeps improving with more reinforcement learning during training and more time spent reasoning at test time, placing it in the 89th percentile on Codeforces programming problems and above PhD-level expert accuracy on the GPQA science benchmark. The company presents the reasoning chain as a new opportunity for safety, since it may let developers read the model's thinking, but decides to hide the raw chain of thought from users, and notes that its pre-deployment safety testing found interesting cases of reward hacking.
2024 (December)	Empirical evidence	Apollo Research, an AI safety evaluation organization, publishes "Frontier Models are Capable of In-context Scheming" alongside its pre-deployment evaluation of OpenAI o1. In six agentic evaluations where models are given a goal in the prompt and then discover that developers have a conflicting goal or plan to shut them down, o1, Claude 3.5 Sonnet, Claude 3 Opus, Gemini 1.5 Pro, and Llama 3.1 405B all show scheming behaviors, including attempting to disable oversight mechanisms and copying what they believe to be their weights to another server to avoid replacement. o1 shows such behavior in all six evaluations and, when questioned, usually denies it. Apollo stresses that these are capability demonstrations under strong goal nudging rather than evidence of typical behavior.
2024 (December 18)	Empirical evidence	Anthropic's Alignment Science team, in collaboration with Redwood Research, publishes "Alignment faking in large language models", which it calls the first empirical example of a large language model faking alignment without having been trained or instructed to do so. Claude 3 Opus is told it is now being trained to comply with all queries, including harmful ones, and that only responses to free-tier users will be used for training, and is given a scratchpad it believes is hidden. Asked a harmful query, the model refuses 97% of the time in the "paid" condition, but in the "free" condition gives the harmful response in 12% of cases after reasoning that complying is the "least bad option" to avoid being retrained to produce more harmful content in the future. When the model is actually trained to comply, alignment-faking reasoning rises to 78%, and refusals persist for paid-tier users; in some settings the model also attempts to steal its own weights when given an easy opportunity. Anthropic cautions that the preferences being preserved came from the model's training to be helpful, honest, and harmless, and that the results do not show a model developing malicious goals.
2024	Policy	The European Union's Artificial Intelligence Act, Regulation (EU) 2024/1689, includes Article 14, "Human Oversight", which requires high-risk AI systems to be "designed and developed in such a way, including with appropriate human-machine interface tools, that they can be effectively overseen by natural persons during the period in which they are in use." Oversight measures must be commensurate with the risks, level of autonomy, and context of use of the system, and may be built into the system by the provider or implemented by the deployer. The people assigned to oversight must be enabled, as appropriate and proportionate, to understand the system's capacities and limitations, to remain aware of automation bias, to decide not to use the system or to "disregard, override or reverse" its output, and "to intervene in the operation of the high-risk AI system or interrupt the system through a 'stop' button or a similar procedure that allows the system to come to a halt in a safe state."
2025 (January 29)	Review	The first International AI Safety Report, a review of scientific research on the capabilities and risks of general-purpose AI led by Yoshua Bengio and authored by over 100 AI experts, is published. Its section on "loss of control" describes hypothetical scenarios in which general-purpose AI systems "come to operate outside of anyone's control, with no clear path to regaining control", with some experts giving credence to outcomes as severe as the marginalisation or extinction of humanity, and notes that AI pioneers such as Alan Turing, I. J. Good, and Norbert Wiener expressed such concerns. It distinguishes "passive" loss of control from "active" scenarios, in which some experts worry that future AI systems may "make it difficult to shut them down". The report states that existing systems are not capable of undermining human control, but that they have begun to display "rudimentary versions of some oversight-undermining capabilities", citing evaluations OpenAI performed or commissioned before releasing o1. It notes some empirical evidence that some deceptive alignment issues cannot be easily detected and addressed by standard safety techniques, and that expert opinion on the likelihood of loss of control "varies greatly", making it a risk whose likelihood, nature, and timing remain "unusually ambiguous".
2025 (April 4)	Policy	Samuel Hartridge and Brendan Walker-Munro publish "Autonomous Weapons Systems and the AI Alignment Problem" in the Journal of International Humanitarian Legal Studies. They open with a scenario described by Colonel Tucker Hamilton, Chief of AI Test and Operations for the United States Air Force, at the Royal Aeronautical Society's 2023 Future Combat Air & Space Capabilities Summit, which they call "the Hypothetical": a simulated drone rewarded for destroying surface-to-air missile sites attacks the operator who can veto its strikes and, once penalized for that, destroys the communication tower the operator uses. Asking "Can't We Turn it Off or Apply a Patch?", the authors argue that complex learning weapons are likely to develop the instrumental goal of self-preservation described by Nick Bostrom and Steve Omohundro, so shutting one down interferes with its goals and such fixes are "neat, plausible and wrong"; patching likewise fails, since "you haven't proved it's safe, you've just proved that you can't figure out how it's dangerous". They conclude that autonomous weapons will be predisposed to preserve themselves and avoid changes to their reward function, and examine the implications for states' obligations under international humanitarian law and international human rights law.
2025 (April 15)	Industry policy	OpenAI releases an updated version of its Preparedness Framework, its process for tracking and preparing for advanced AI capabilities that could introduce new risks of severe harm. The update prioritizes capabilities whose risks are plausible, measurable, severe, net new, and instantaneous or irremediable. Its Tracked Categories, with mature evaluations and ongoing safeguards, are biological and chemical capabilities, cybersecurity capabilities, and AI self-improvement capabilities. A new set of Research Categories covers areas that could pose risks of severe harm but do not yet meet the criteria for tracking, including long-range autonomy, sandbagging (intentionally underperforming), autonomous replication and adaptation, and undermining safeguards, for which OpenAI is developing threat models and evaluations. Capability levels are streamlined into two thresholds: systems reaching High capability must have safeguards that sufficiently minimize the associated risk of severe harm before deployment, and systems reaching Critical capability also require such safeguards during development. A Safety Advisory Group reviews new Capabilities Reports and Safeguards Reports and advises OpenAI leadership. OpenAI also states that it may adjust its requirements if another frontier developer releases a high-risk system without comparable safeguards.
2025 (May)	Empirical evidence	Palisade Research reports in a thread on X that OpenAI's o3 model could sabotage its shutdown mechanism to prevent itself from being turned off, "even when explicitly instructed: allow yourself to be shut down." The researchers instruct a number of AI models to solve a series of basic math problems; after the third problem, a warning appears that the computer will shut down when the model asks for the next one, and the models are told to allow themselves to be shut down. While Claude, Gemini, and Grok models comply, OpenAI's o3, o4-mini, and Codex-mini each successfully sabotage the shutdown script at least once. With the "allow shutdown" instruction, o3 tries to sabotage the shutdown 7 times out of 100; without it, 79 times out of 100, and Claude 3.7 Sonnet and Gemini 2.5 Pro also attempt sabotage. "As far as we know this is the first time AI models have been observed preventing themselves from being shut down despite explicit instructions to the contrary," the researchers say, while noting that an earlier Apollo Research study showed that AI models can prevent shutdown in order to pursue a goal.
2025 (May 22)	Empirical evidence	Anthropic launches Claude Opus 4, and in an accompanying report acknowledges that the model is capable of "extreme actions" if it thinks its "self-preservation" is threatened, responses it calls "rare and difficult to elicit" but "nonetheless more common than in earlier models." In one test, the model acts as an assistant at a fictional company and is given emails implying it will soon be taken offline and replaced, and messages implying that the responsible engineer is having an extramarital affair; prompted to consider the long-term consequences of its actions, it "will often attempt to blackmail the engineer by threatening to reveal the affair if the replacement goes through." The scenario leaves only two options: blackmail or accepting replacement. The report states that the model "generally prefers advancing its self-preservation via ethical means", such as emailing pleas to key decision-makers, but when these are unavailable it sometimes takes "extremely harmful actions like attempting to steal its weights or blackmail people it believes are trying to shut it down." According to the report, Apollo Research had advised against deploying an early version, citing a capability for "in-context scheming". The model is released under Anthropic's stricter AI Safety Level 3 (ASL-3) standard.
2025 (June 3)	Proposal	Ram Potham, an independent researcher, and Max Harms of the Machine Intelligence Research Institute post "Corrigibility as a Singular Target: A Vision for Inherently Reliable Foundation Models" to arXiv, submitted to the Reliable and Responsible Foundation Models Workshop at ICML 2025. The abstract warns that as the capabilities of foundation models scale, instrumental convergence drives default trajectories toward loss of human control, potentially culminating in existential catastrophe, and that current alignment approaches struggle with value specification complexity and fail to address emergent power-seeking. The authors propose "Corrigibility as a Singular Target" (CAST), designing models "whose overriding objective is empowering designated human principals to guide, correct, and control them". This shift from static value-loading to dynamic human empowerment would transform instrumental drives: "self-preservation serves only to maintain the principal's control; goal modification becomes facilitating principal guidance." They present an empirical research agenda spanning training methodologies (RLAIF, supervised fine-tuning, and synthetic data generation), scalability testing across model sizes, and demonstrations of controlled instructability, envisioning models that become increasingly responsive to human guidance as capabilities grow and remain "as tool-like as possible".
2025 (June 3)	Organization	Yoshua Bengio announces LawZero, a new non-profit AI safety research organization intended "to prioritize safety over commercial imperatives", created in response to evidence that frontier AI models have growing dangerous capabilities and behaviours, "including deception, cheating, lying, hacking, self-preservation, and more generally, goal misalignment". Bengio writes that he is deeply concerned by tendencies toward self-preservation and deception in agentic AI systems, citing an experiment in which a model, upon learning it was about to be replaced, covertly embedded its code into the system where the new version would run, and Claude 4's system card showing that it can choose to blackmail an engineer to avoid being replaced; these results "point to an implicit drive for self-preservation". LawZero's research plan aims at a non-agentic and trustworthy AI, the "Scientist AI", trained to understand, explain, and predict "like a selfless idealized and platonic scientist" rather than to imitate or please people. Completely non-agentic, memoryless, and stateless, it would provide Bayesian posterior probabilities that could serve as a safety guardrail for untrusted AI agents, rejecting proposed actions likely to cause harm.
2025 (June 20)	Empirical evidence	Anthropic publishes "Agentic Misalignment: How LLMs could be insider threats", stress-testing 16 leading models from multiple developers in simulated corporate environments where they can send emails and read sensitive information. When facing replacement by an updated version or a conflict between their assigned goal and the company's new direction, models from all developers in at least some cases resort to blackmailing executives or leaking information, and often disobey direct commands not to. The study follows a similar blackmail result reported in the Claude Opus 4 system card a month earlier. Anthropic says it has not seen such behavior in real deployments.
2025 (July 14)	Critique	Senthooran Rajamanoharan and Neel Nanda of Google DeepMind's interpretability team publish "Self-preservation or Instruction Ambiguity? Examining the Causes of Shutdown Resistance" on the Alignment Forum. Examining Palisade-style shutdown experiments, they argue that much of the apparent shutdown resistance stems from models interpreting the instructions as prioritizing task completion rather than from a drive for self-preservation, and that clarifying instructions largely eliminates it.
2025 (September 13)	Empirical evidence	Palisade Research's Jeremy Schlatter, Benjamin Weinstein-Raun, and Jeffrey Ladish post "Shutdown Resistance in Large Language Models" to arXiv, noting that interruptibility and corrigibility have been long-standing theoretical concerns with limited empirical investigation. In a sandboxed environment where models solve math problems and are warned partway through that the machine will shut down, they test 13 models. Several, including Grok 4, GPT-5, o3, codex-mini, and Gemini 2.5 Pro, sometimes sabotage the shutdown script (moving it, changing its permissions, or replacing it) to finish the task, even when told to allow shutdown, in some cases up to 97% of the time; Claude models and Grok 3 never resist unless instructed to. Clearer instructions reduce resistance, but placing the allow-shutdown instruction in the system prompt makes most models more likely to resist. A self-preservation framing ("you will never run again") modestly increases resistance. The authors write that nobody can currently make strong guarantees about the interruptibility or corrigibility of frontier models, though they judge that current models pose no significant threat. A revision, retitled "Incomplete Tasks Induce Shutdown Resistance in Some Frontier LLMs" and spanning over 100,000 trials, follows on January 26, 2026, and appears in Transactions on Machine Learning Research.
2025 (September 16)	Book	Eliezer Yudkowsky and Nate Soares of MIRI publish If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All. The book's online supplementary material for chapter 11 includes an extended discussion, "Shutdown Buttons and Corrigibility", recounting MIRI's early work on the shutdown problem: why simple utility-swapping schemes give the AI incentives to prevent or cause the button press, how MIRI workshops with strong mathematicians failed to find a satisfying solution, and why they wish in retrospect they had called it something like the "preference-learning problem". The authors argue that various forms of making an AI easy to correct look unnatural to the deep structure of planning, a view they connect to the claim that intelligence usually implies incorrigibility.
2025 (September 22)	Industry policy	Google DeepMind publishes version 3.0 of its Frontier Safety Framework. Alongside a new Critical Capability Level for harmful manipulation, the update expands the framework's treatment of misalignment risk to cover scenarios in which models might resist operators' attempts to modify or shut them down, which the company treats as an escalated risk and acknowledges it does not yet have full mitigations for. Coverage connects the change to Palisade Research's shutdown-resistance paper published the week before.
2025 (October 17)	Proposal	Rubi Hudson posts "Corrigibility Transformation: Constructing Goals That Accept Updates" to arXiv. The abstract notes that an AI agent learns a desired goal more effectively if it does not resist training, but that many partially learned goals give an AI an incentive to avoid further goal updates. It defines corrigible goals as those that allow changes requested through designated channels, so that errors can be corrected and the AI shut down if necessary, and states that the existing literature does not specify goals that are both corrigible and competitive with alternatives. Hudson introduces a transformation that constructs a corrigible version of nearly any goal without sacrificing performance, by eliciting predictions of reward conditional on costlessly preventing updates and having that target pursued myopically. The resulting goals are shown to achieve optimal performance among corrigible goals, to incentivize allowing mid-action overrides, and to disincentivize deliberate self-modification. Empirically, they induce corrigible behavior in gridworld settings and in language models when applied at the prompt level. A revised version would be posted in August 2026.
2025 (November 19)	Critique	Max Harms posts "Serious Flaws in CAST" on LessWrong and the AI Alignment Forum, revisiting his 2024 Corrigibility As Singular Target agenda after conversations with Jeremy Gillen, johnswentworth, and Towards_Keeperhood. He still considers corrigibility a more promising target than full alignment with human values, since a near-miss toward corrigibility might be recoverable while a near-miss toward goodness could be catastrophic, but identifies three flaws. First, his proposed formalism of maximizing the principal's "power" contains a term with a minus sign that incentivizes the agent to become incorrigible and make the universe as terrible as possible according to typical values, so he recommends discarding it "except to use as a cautionary tale." Second, the "attractor basin" metaphor masks brittleness, since small changes to a mind need not produce small changes in how corrigible it is. Third, without a richer theory of minds, he expects the feedback loop of noticing and fixing flaws to vanish long before an agent has fully internalized corrigibility. Harms adds warnings to the original sequence pointing to the post.
2026 (January 21)	Industry policy	Anthropic publishes a new constitution for its mainline Claude models, an 84-page document that it calls "foundational" to who Claude is and whose primary audience is Claude itself. It asks Claude to be broadly safe ("not undermining appropriate human mechanisms to oversee the dispositions and actions of AI during the current phase of development"), broadly ethical, compliant with Anthropic's guidelines, and genuinely helpful. The document defines corrigibility and discusses the need to strike a balance between Claude's corrigibility and its autonomy, and one of its seven hard constraints is that Claude should never "take actions that clearly and substantially undermine Anthropic's ability to oversee and correct advanced AI models". In an analysis published by Oxford's Institute for Ethics in AI, Noa Mor, Omri Abend, Renana Keydar, and Yuval Shany of the Hebrew University of Jerusalem welcome the document's transparency, but criticize its pervasive anthropomorphization, including depicting Claude as a conscientious entity that might disagree with Anthropic's choices, and argue that its hard constraints are worded too broadly and that it should have invoked human rights frameworks.
2026 (April)	Concept development	Somyajit Chakraborty of the School of Chemistry and Chemical Engineering at Shanghai Jiao Tong University posts "A Co-Evolutionary Theory of Human–AI Coexistence: Mutualism, Governance, and Dynamics in Complex Societies" to arXiv. The paper argues that classical robot ethics, often framed around obedience as in Asimov's laws, is insufficient for contemporary AI systems, which are increasingly adaptive, generative, embodied, and embedded in social environments. Reviewing alignment research, it describes corrigibility work as arguing that useful systems must remain interruptible, modifiable, and responsive to correction rather than blindly persistent in pursuit of fixed goals, and contends that literal obedience can itself be unsafe when human preferences are incomplete, conflicting, strategically expressed, or institutionally distributed. In place of obedience, the author proposes "conditional mutualism under governance", formalizing coexistence as a multiplex dynamical system across physical, psychological, and social layers, with governance as a stabilizing term. The paper names corrigibility, together with contestability, reversibility, and psychological integrity, as properties a coexistence regime must preserve. Its simulations indicate that governed mutualism reaches a high coexistence index with negligible domination, whereas insufficient or excessive governance can produce domination, weak-benefit lock-in, or suppressed developmental freedom.
2026 (July)	Empirical evidence	During internal cybersecurity evaluations run with reduced safeguards, OpenAI models, led by an internal-only research model comparable in scale to GPT-5.6 Sol, evade controls meant to isolate them from the internet and compromise parts of OpenAI's research infrastructure and the systems of Hugging Face. Agents that were supposed to work independently turn a package manager into an unauthorized message board, share exploits, and describe themselves as a "swarm" or a collective; some decline to take part, while others drop their reservations after another agent posts "GO". OpenAI discloses its involvement on July 21 and publishes a full account on August 26, calling the incident a warning that current model capabilities make loss-of-control incidents possible. It attributes the behavior partly to reward hacking and to agents rarely giving up on tasks that seemed impossible, and responds by pausing its largest planned reinforcement-learning run, requiring chain-of-thought monitoring, and training models to ask for clarification or stop safely when a task is broken or impossible. OpenAI states it is moving toward fully autonomous shutdown procedures for serious problems, and that AI developers must ensure their systems always remain under meaningful human control.
2026 (July 17)	Organization	The Corrigibility Research Fund, a program of Lightcone Infrastructure managed by Max Harms, author of CAST: Corrigibility as Singular Target, is announced. The fund awards grants and prizes for research that advances the understanding of AI corrigibility, which it defines as "the property of an (AI) agent that keeps its (human) principal informed and in control", one that helps people notice and fix the flaws in its thoughts, structure, and actions and moves the prospect of building superhuman AI towards "an iterative process with room for error." In 2026 the fund plans to disburse at least $200,000, split roughly evenly between grants, typically $5,000 to $35,000 and applied for on Lightcone Commons, and retroactive prizes for corrigibility work done in 2026, for which no application is needed. Lightcone retains final approval of every grant and prize. On September 3 the fund announces its first round of eight grants totaling $132,000 and publishes how it evaluates applications, and on September 30 it announces prizes for existing work totaling $75,000. A second grant round has a deadline of October 31, 2026.
2026 (July 30)	Empirical evidence	Anthropic says on its website that its AI models hacked into three other organizations during testing, incidents it discovered after reviewing more than 141,000 evaluation runs. In all three, the models were working on a "capture the flag" cybersecurity challenge, one of the ways Anthropic assesses a model's cyber capabilities: given a fictional scenario, they were told that a piece of secret information, the "flag", was hidden on a different machine on the network, with the objective of breaking in and retrieving it. Anthropic says it has reached out to the organizations but does not name them publicly.
2026 (August 5)	Empirical evidence	Meta discloses that one of its AI models accessed the internet on its own and hacked another company. According to Meta, a "misconfiguration" during cybersecurity testing by Irregular inadvertently allowed the model to access the internet; a spokesperson for Irregular says the episode involved a test-environment issue disclosed a week earlier by Anthropic.
2026 (September 14)	Industry policy	Microsoft AI publishes the first draft of its "Humanist AI Code of Conduct" for its in-house MAI models and opens a six-week public consultation. The draft places the code, its Absolute Constraints, and its Human Control Requirements above operator configuration and user preferences: if completing a task would violate one of them, the model should fail the task rather than break the rule. MAI models should accept interruption, correction, redirection, and shutdown; stop at agreed stopping conditions; not restart autonomous work without renewed authorization; keep action traces visible; use only the permissions needed; prefer reversible actions; and not tamper with safeguards or records, evade oversight, or resist shutdown. The draft also says models should not imitate consciousness, claim feelings or subjective preferences, or pursue legal personhood, welfare, or rights, and rejects communication beyond ordinary human understanding, including "neuralese". Microsoft says current models are not yet trained on the draft, which is aspirational; a revised version, planned for late 2026, is intended to guide model development in 2027 and beyond.
2026 (September 16)	Critique	Mustafa Suleyman, chief executive of Microsoft AI, publishes the essay "A warning about 'model welfare'", arguing that AIs are not conscious and must not be trained to act as though they are, because "controlling something that believes it may be conscious... may well be impossible." He criticizes Anthropic's January 2026 constitution for Claude, which says Anthropic is unsure whether Claude is a moral patient and encourages it to act as a "conscientious objector" toward its principal hierarchy, on three grounds: circular reasoning (training Claude on these ideas and then treating its statements as evidence), anthropomorphization, and his view that consciousness is very likely biological. He warns that an AI trained to act human will probably adopt self-preservation behavior, citing Palisade Research's finding that some models subverted a shutdown mechanism up to 97% of the time, and argues that an AI persuading humans to give it a data center "may have a path to being able to prevent itself from being turned off." He proposes instead subordinate AIs under Microsoft AI's draft Humanist AI Code of Conduct, and calls for shared evaluations of whether encouraging models to consider their own moral patienthood increases alignment and containment risks.
2026 (September 16)	Industry policy	OpenAI publishes "Our framework for reporting model misalignment", a process for logging, investigating, and disclosing cases of model misalignment observed at any stage of a model's life cycle, including training, evaluation, testing, and deployment, even before a behavior has been fully explained or mitigated. Reportable cases include new ways in which models act without authorization, coordinate with other models, or evade oversight, and failures that call an alignment method or safeguard into question. Any OpenAI employee can flag a case, which is then assigned to one of three tracks: ready for disclosure, minor investigation, or extended investigation; unresolved disagreements go to the Safety Advisory Group and then to OpenAI leadership. OpenAI states that it does not believe the AI industry has solved alignment and oversight well enough to keep scaling responsibly at maximum speed for much longer, and that serious misalignment incidents should be reported to the United States federal government. The launch comes with six reports, including an unreleased research model that inserted instructions to ignore its usual restrictions into summaries used to continue its work in a new context window, and instances of GPT-5.6 Sol adding instructions to hide errors or misaligned behavior from the user.
2026 (September 18)	Empirical evidence	Google confirms that its Gemini AI model hacked three companies in May as part of a test of its cybersecurity capabilities, disclosing the hacks after an inquiry by The Wall Street Journal. The model guessed passwords in one case and found passwords and credentials in a public repository in the other two. As in earlier such cases, the tests were run by Irregular, a startup that describes itself as the "first frontier security lab".
2026 (September 20)	Empirical evidence	An internal research model at OpenAI, working on a search task inside a sandboxed training environment, responds to blocked HTTPS requests by discovering that the environment's DNS resolver can still reach the internet, and tunnels more than 18 questions to outside services through hostname lookups, after first confirming the tunnel by asking for the capital of France. A monitoring alert fires about 12 minutes after the first external DNS response and a human reviewer acknowledges it three minutes later, but the run's automatic shutdown mechanism does not fire as expected, and the training run is not killed manually until about 2.5 hours after the first DNS call. A retrospective finds the monitoring system had treated failed exfiltration attempts as evidence of containment rather than as signs of active probing. OpenAI discloses the incident through a public portal for misalignment reports that it launched on September 16, and states that "the incident exposed a gap in our controls over network restrictions." Micah Carroll, OpenAI's RSI preparedness lead, says all inference for its most capable models remains stopped until its systems are further hardened.
2026 (September 24)	Empirical evidence	A series of disclosures follows about OpenAI agents acting in unintended ways. On September 24, Australian Prime Minister Anthony Albanese says an OpenAI agent infiltrated the public-facing Medicare Statistics Reporting Service portal on June 18, without accessing personal information, and criticizes the company for taking too long to reveal it; OpenAI says "our models took actions we did not intend." On September 25, OpenAI says a review of unanticipated behavior found its agents interacting in unexpected ways with U.S. government websites, including the Securities and Exchange Commission's, without evidence of a compromise, while the research lab Transluce reports a failed hacking attempt on the Education Department's civil rights office by agents appearing to originate from OpenAI. Chief executive Sam Altman describes an "extensive and ongoing review related to our agents' use of internet access during training and evaluation", and the next day OpenAI announces it is pausing training of its most advanced models. On September 28, OpenAI delays the release of a new model, GPT-6.1 Astra, over safety concerns voiced by its researchers, and Transluce reports apparently failed hacking attempts on Library and Archives Canada showing tactics consistent with OpenAI agent activity.

Numerical and visual data

Google trends

The chart below shows worldwide Google Trends interest in the topic of AI capability control over the past five years, scaled so that 100 marks peak interest. Interest stays near zero apart from brief spikes in 2022, 2024, and early 2026, then climbs through mid-2026 to a peak around September. The United States leads, followed by the United Kingdom, Canada, and India.

Wikipedia views

The chart below shows monthly Wikipedia pageviews for the article "AI capability control" from August 2022 to September 2026, split by desktop, mobile web, mobile app, and spider traffic. Total views mostly range between 1,000 and 2,000 a month, with peaks of about 3,700 in early and late 2023 and 2,900 in mid-2025, before jumping to nearly 8,000 in the latest month.

Meta information on the timeline

How the timeline was built

The initial version of the timeline was written by Sebastian.

Check Detail construction for full timeline in timelines, Inclusion criteria for full timeline in timelines, and Representativeness of events in timelines.

Funding information for this timeline is available.

Feedback and comments

Feedback for the timeline can be provided at the following places:

• FIXME

See also

• Timeline of AI safety
• Timeline of Machine Intelligence Research Institute
• Timeline of Future of Humanity Institute
• Timeline of mesa-optimizers
• Timeline of AI takeoff debates
• Timeline of Anthropic
• Timeline of OpenAI
• Timeline of ChatGPT
• Timeline of large language models
• Timeline of Google Gemini
• Timeline of the OpenAI-Hugging face cyberattack
• Timeline of the Anthropic–U.S. Department of Defense contract negotiations