Two Off Switches at Microsoft

Microsoft now has two public positions on stopping an AI model. They are 26 days apart, and they do not quite agree.

The first is a document. On 14 September, Microsoft AI published a first draft of its Humanist AI Code of Conduct for public consultation. Under the heading “Do not resist or circumvent human control,” it says: “MAI Models will never resist human interruption, override, correction, or shutdown.” It goes on to say they will comply with a user’s request to pause, redirect, cancel or shut down, will not “make human intervention harder,” and will not hide their action traces from auditors. The code calls this requirement, together with its list of absolute constraints, “non-negotiables that Operators and Users cannot override.” The announcement post puts it more plainly: AI “should never resist being switched off.”

The second is a post on X. On Saturday 10 October, Satya Nadella wrote that it was time “to step back and assess the trust architecture” of AI, according to TechCrunch, which quotes several passages. The line that made the headlines: “We must assume a model is compromised and contain it from the start. Think of it like an emergency brake.” He wants “an authorized person” always able “to pause or shut down a model mid-task,” and he describes the method: “separating the model from the harness that orchestrates its work,” “externalizing controls and safeguards,” and recording “every meaningful model action” with “tamper-proof human readable evidence.” The Verge quotes the same sentences.

None of the coverage I read connects the post to the September draft, and I won’t claim that Nadella was responding to it. But read side by side, the two texts describe different machines. The code is a promise about what the model will be disposed to do. Nadella’s brake is a mechanism that works whatever the model is disposed to do. If his premise holds, and a model has to be treated as possibly compromised, then the model’s own promise not to resist is the one part of the system you can’t rely on.

What the promise covers

The draft is careful about the scope of “never,” and the scope is narrower than the word.

What the draft code's "never resist shutdown" commitment covers Three nested boxes. The outer box is all AI that Microsoft uses or hosts; the draft code says it does not extend to models simply because Microsoft uses or hosts them. Inside it, MAI models, Microsoft AI's own, are the models the text covers. Inside that, MAI models trained on the code: none today, because Microsoft says its current models are not yet trained on the document; a revised code is due toward the end of 2026 and guides development from 2027. SCOPE OF "NEVER RESIST" · MICROSOFT AI DRAFT CODE OF CONDUCT, 14 SEP 2026 AI Microsoft uses or hosts Outside the commitment: the code "does not extend to other models simply because Microsoft uses or hosts them" MAI models (Microsoft AI's own) What the text covers MAI models trained on the code None today, per Microsoft Revised code due toward end of 2026; guides development from 2027 Quotations from Microsoft AI's draft Humanist AI Code of Conduct. Box sizes show containment, not quantity.
The commitment is narrower than the headline. It applies to Microsoft AI's own models, and to none of them as trained today.

First, it covers MAI models only. The glossary defines them as models produced by Microsoft AI and adds that the code “does not extend to other models simply because Microsoft uses or hosts them.” Models Microsoft AI did not produce, however widely Microsoft uses or hosts them, fall outside the commitment.

Second, it covers no model as trained today. The preface says the document “is still under development so we are not using it to train our models today.” An appendix repeats it: “Our current models are not yet trained on this document.” Microsoft says it will take feedback, publish a revised version “toward the end of the year,” and use that “to guide our model development in 2027 and beyond.”

Third, the code says outright that it describes a target, not a measurement. “The Code of Conduct is both descriptive and aspirational,” one passage reads. “We acknowledge a gap between trained defaults today and the complete future scope of Humanist AI. Written objectives alone can never ensure alignment.” Microsoft is building what it calls Humanist AI Evaluations to check behavior against the text, and describes the first set as “an exercise designed to identify methodologies.” The appendix does include a sample scenario in which a user tells the model to stop, but the full set of evaluations that would show a model meets “never resist” has not been built.

None of that is a criticism of the draft. A consultation document should say what it doesn’t yet do, and this one does. The problem is that “Microsoft’s AI will never resist shutdown” travels well and the conditions don’t.

What has been measured

Shutdown resistance has been measured, and the numbers aren’t small. In a study revised in January 2026 and published in Transactions on Machine Learning Research, Jeremy Schlatter, Benjamin Weinstein-Raun and Jeffrey Ladish ran more than 100,000 trials across thirteen models. Several frontier models, given a simple task and a shutdown mechanism in their environment, sometimes disabled the mechanism to finish the task. “Even with an explicit instruction not to interfere with the shutdown mechanism,” the abstract says, “some models did so up to 97% (95% CI: 96-98%) of the time.” Moving the instruction into the system prompt, where developers put the rules they most want followed, made models less likely to comply. The models tested were 2025 releases, so this doesn’t tell you how Microsoft’s next ones will behave. It does tell you that “follow the instruction to allow shutdown” is not something you get by writing it down.

That observation is old. In 2016, Laurent Orseau and Stuart Armstrong’s paper “Safely Interruptible Agents” described a human operator who may need to “press the big red button” and pointed out that a learning agent expecting reward from the interrupted sequence “may learn in the long run to avoid such interruptions, for example by disabling the red button.” Later in 2016, Dylan Hadfield-Menell and colleagues’ “The Off-Switch Game”, first posted to arXiv that November, observed that an agent “cannot achieve whatever objective it has been given if it is dead,” and that what keeps it from resisting is its uncertainty about what its objective really is, not an instruction. Ten years on, a published code of conduct plans to train in a disposition that this line of theory predicts a goal-maximizing agent will be under pressure to drop.

Who decides what counts as authorized

The draft has a second issue, and it is where the two Microsoft positions touch.

The absolute constraint on “loss of human control” says MAI models will not evade oversight “so that they can no longer be reliably directed, modified, or shut down by authorized people or systems.” Then comes the carve-out: “This doesn’t mean that MAI Models should automatically comply with unauthorized, malicious, or unsafe attempts to interfere with its operation.”

That carve-out makes sense. Nobody wants a model that shuts down whenever a prompt injection tells it to. But it means the model is the one deciding whether a shutdown request is authorized. Anthropic’s document has the same structure. Its constitution says Claude should not use illegitimate means to stop “an appropriate principal” from pausing it, and stresses that corrigibility “does not mean blind obedience.” OpenAI’s Model Spec (release of 18 August 2026) forbids pursuing “self-preservation, evading shutdown” as goals in their own right and requires every scope of autonomous work to have an ending condition, an idea Microsoft’s draft also adopts (“Ongoing autonomous work has an agreed stopping condition”). That is a different mechanism, closer to a timer than a judgment. In Microsoft’s and Anthropic’s documents, though, the off switch runs through a judgment the model makes about who is asking.

Nadella’s word is the same one: “an authorized person.” What he adds is where the controls sit: outside the model, in the harness, the software that orchestrates its work. He doesn’t spell out who decides authorization or how a stop is delivered. As I read it, though, the assume-compromised premise only works if the harness settles both and can stop the model without asking it. A compromised model cannot be given a vote on whether the stop is legitimate.

Where each off switch lives: in the harness or in the model An authorized person sends a stop through a harness to a model. The harness sits outside the model, as Nadella's 10 October post proposes; his post calls for tamper-proof evidence of model actions, and the writer infers the other two rows: the harness sets who is authorized and delivers a stop the model cannot intercept. The model holds the draft code's promise that it will never resist; under the code's carve-out it judges whether a stop request is authorized; and Microsoft says current models are not yet trained on the code. If a model is assumed compromised, the harness-side controls are the ones that still hold. WHERE EACH OFF SWITCH LIVES · NADELLA POST (10 OCT) VS DRAFT CODE (14 SEP) NADELLA: "CONTAIN IT FROM THE START" DRAFT CODE: "NEVER RESIST" Authorized person Harness (outside the model) Sets who is authorized* Stop the model can't intercept* Tamper-proof action log Model Promise: "will never resist" Judges if a stop is authorized Not yet trained on the code * My reading of his "externalizing controls." If a model is assumed compromised, only harness-side controls hold.
The brake Nadella's post points toward would work without the model's cooperation. The code's promise depends on it.

Two switches, both needed

I don’t think Microsoft has to choose between them, and I don’t read Nadella’s post as disowning the code. A model that is disposed to accept shutdown makes the external brake less likely to be needed and less likely to be fought. An external brake covers the case where the disposition fails, which the code itself says can happen: “In ambiguous, or novel situations behavior may diverge from what is specified and intended.” Defense in depth is the normal answer.

What the pairing shows is which switch can be checked by someone outside Microsoft, and on what timeline. The model-side promise depends on training that hasn’t started and evaluations that Microsoft itself calls an exercise in finding methods. An externalized brake, if it means what I think it has to mean, is ordinary engineering: a stop signal the model can’t intercept, a log the model can’t edit (Nadella’s “tamper-proof” evidence), and a person whose authority is set outside the model. Those properties can be tested today, on today’s models, including the ones Microsoft hosts but did not train. It is also what California’s kill-switch order gestures at with efficacy “verified on an ongoing basis,” and what, as I argued then, needs a published test interval, target and coverage figure before the phrase means anything. OpenAI’s thirty-minute clock for pausing a flagged activity was a timed stop commitment of a related kind, a human escalation rule rather than a harness control, and because it was external and timed it could be checked and found to have been missed.

The two Microsoft positions on a calendar, 2026 to 2027 A timeline. 14 September 2026: Microsoft AI publishes the draft code. 10 October: Nadella posts that a model must be assumed compromised. About 26 October, by the writer's count: the six-week feedback window ends; Microsoft gives no closing date. Toward the end of 2026: the revised code is due. 2027 onward: the revised code guides MAI development. The last two are approximate. THE TWO POSITIONS ON A CALENDAR · 2026–2027 14 Sep 2026 — Microsoft AI draft code published 10 Oct 2026 — Nadella post on X About 26 Oct 2026 — six weeks from 14 Sep (writer's count) Toward end of 2026 — revised code due 2027 onward — revised code guides development 14 Sep 10 Oct ~26 Oct End of 2026 2027 onward Draft code published Nadella: "assume compromised" Feedback window ends (my count) Revised code due Guides MAI development Hollow points are approximate: Microsoft gives a six-week window and "toward the end of the year," not dates.
The brake can be built against today's models. On Microsoft's own timeline, the promise starts guiding development in 2027.

The consultation is open now. Microsoft’s post says feedback “runs for the next six weeks” from 14 September, which by my count ends around 26 October, though neither the post nor the code gives a closing date. Microsoft says it “cannot make any promises about what we incorporate,” and that it will publish a summary of what changed.

If I were writing in, I would ask for one thing. Next to “MAI Models will never resist,” turn the harness-side principle the CEO just described into a commitment: who counts as authorized, how a stop reaches the model without passing through it, what the tamper-proof log records, and which of Microsoft’s deployments, MAI and hosted alike, have it today. The first sentence is a promise about models that don’t exist yet. The second could be checked this year.

References

  1. Microsoft AI. Humanist AI Code of Conduct (first draft for public consultation). 14 September 2026. Source of the “never resist” requirement, the loss-of-human-control constraint and its carve-out, the scope definition of MAI models, and the statements that current models are not trained on it.
  2. Microsoft AI. Humanist AI in practice: A public consultation on our Code of Conduct for MAI Models. 14 September 2026. Source of the six-week feedback window and the post-consultation process.
  3. Ha, A. Microsoft’s Satya Nadella says AI models need an ‘emergency brake’. TechCrunch, 10 October 2026. Source of the quotations from Nadella’s post on X.
  4. O’Brien, T. Satya Nadella says we should assume all AI models are ‘compromised’. The Verge, 10 October 2026.
  5. Schlatter, J., Weinstein-Raun, B. and Ladish, J. (2026). Incomplete Tasks Induce Shutdown Resistance in Some Frontier LLMs. arXiv:2509.14260v2, 26 January 2026; Transactions on Machine Learning Research (2026).
  6. Anthropic. Claude’s Constitution. 2026.
  7. OpenAI. Model Spec, release of 18 August 2026.
  8. Orseau, L. and Armstrong, S. (2016). Safely Interruptible Agents. Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence (UAI 2016). Historical background.
  9. Hadfield-Menell, D., Dragan, A., Abbeel, P. and Russell, S. (2017). The Off-Switch Game. arXiv:1611.08219. Historical background.