Newsom's Kill Switch Order Does Not Require a Kill Switch. It Requires a Memo by November 16.

Gavin Newsom signed Executive Order N-9-26 on September 18. The press release from his office is headlined “Governor Newsom issues executive order to accelerate independent oversight and advance the creation of an AI kill switch,” and most of the coverage since has taken the second half of that phrase at face value.

The order does not create a kill switch. It asks whether one is possible. The operative paragraph gives the Government Operations Agency until November 16, 2026, fifty-nine days after signing, to send the Governor’s office recommendations “addressing the technical feasibility and potential efficacy of amendments to existing state laws regarding AI safety and security,” and then lists four candidate amendments. The kill switch is the third.

Scanned text of the executive order listing four candidate amendments labelled a through d, followed by two paragraphs directing that the order be filed with the Secretary of State and stating that it creates no enforceable rights.
The four amendments the Government Operations Agency has been asked to evaluate, and the two paragraphs that close the order. Item (c) is the kill switch. The final paragraph is the one that decides what the other four are worth today. Image: Executive Office of the Governor of California, Executive Order N-9-26, signed 18 September 2026. Cropped from page 3 of the signed PDF; reproduced for commentary.

That last paragraph is not boilerplate I am quoting to be clever. It reads: “This Order is not intended to, and does not, create any rights or benefits, substantive or procedural, enforceable at law or in equity, against the State of California, its agencies, departments, entities, officers, employees, or any other person.” Nothing in the order binds a frontier developer to anything. A kill-switch mandate would need the Legislature, and the Legislature would need the November memo first.

What the order actually moves

Two of its three paragraphs do bind something, and they got almost none of the attention. They pull forward deadlines from two bills Newsom had signed nine days earlier, on September 9.

Senate Bill 813, Chapter 179, created Government Code section 8898.1, which opens “On or before January 1, 2028, the agency shall do all of the following:” and then lists the machinery for designating what the statute calls independent verification organizations: application requirements, designation criteria, suspension procedures. The executive order moves that to May 1, 2027. Assembly Bill 1405, Chapter 178, created section 11549.82, whose subdivision (a) begins “No later than January 1, 2029” and requires an AI Auditor Registry, a fee schedule, and a public mechanism for reporting auditor misconduct. The order moves that to December 1, 2027.

Statutory deadlines versus the deadlines Executive Order N-9-26 substitutes Three deadlines on a timeline running from September 2026 to early 2029. Designation criteria for independent verification organizations: the statutory deadline under SB 813 is 1 January 2028; the executive order requires 1 May 2027, eight months earlier. The AI Auditor Registry: the statutory deadline under AB 1405 is 1 January 2029; the executive order requires 1 December 2027, thirteen months earlier. The recommendations memo on feasibility, including the kill switch, is due 16 November 2026 and has no statutory predecessor. DEADLINES · EO N-9-26 VS THE UNDERLYING STATUTES Jan 2027 Jan 2028 Jan 2029 Designation criteria, IVOs AI Auditor Registry Feasibility memo 1 January 2028 — statutory deadline, SB 813 Ch. 179, Gov. Code 8898.1 1 January 2029 — statutory deadline, AB 1405 Ch. 178, Gov. Code 11549.82(a) 1 May 2027 — deadline set by Executive Order N-9-26 1 December 2027 — deadline set by Executive Order N-9-26 16 November 2026 — recommendations due, no statutory predecessor 8 months earlier 13 months earlier 16 Nov 2026 — no statutory predecessor Executive order Statute Both statutes were signed 9 September 2026, nine days before the order.
The binding part of the order is a calendar change to machinery the Legislature had already authorised. The kill switch is the bottom row, and the bottom row is a memo.

So the order accelerates the scaffolding for independent verification by eight and thirteen months respectively, and asks, separately, what that verification should be pointed at. Read in that order it is a more coherent document than the headline suggests. It is also a much smaller one.

One clause, one phrase

Here is the clause the whole thing turns on, in full: “Requiring the creation of a ‘kill switch’ for frontier models, with the efficacy of the switch verified on an ongoing basis by an independent verification organization.”

Note the phrase. Not “routinely”: that word appears nowhere in the order. Its nearest cousin is “periodic audits and evaluations,” in the clause about embedding auditors onsite. The kill switch gets “on an ongoing basis,” and the order says nothing about by what method, at what interval, or against what threshold.

That is not a drafting failure. Specifying the method is exactly what the November memo is for. The question is whether anyone has a method to hand over.

The field that already does this

There is one body of engineering practice that has spent forty years verifying that shutdown devices still work: functional safety, as codified in IEC 61508 and, for the process industries, IEC 61511. A safety instrumented function (sensor, logic solver, final element) gets a written proof-test procedure, and the interval between those tests is not chosen by taste. It is derived from a target probability of failure on demand: the number that says how likely the device is to be dead at the moment you finally reach for it. “Ongoing verification” in that world resolves to an interval, and the interval is traceable to a risk target.

Which is the good news for anyone drafting clause (c). The bad news is in the same equation.

The derivation assumes the proof test finds every dangerous failure the automatic diagnostics missed. It does not. The gap has a name: proof test coverage. In a 2014 worked example, William Goble of exida put numbers to it. Take a product with 100 FITS of dangerous failures, where FITS counts failures per billion hours. Poor automatic diagnostics catch 10 of them. A manual proof test run during operation catches 72 of the remaining 90. “The proof test coverage is 72/90 = 80%,” Goble writes, and then: “There are 18 FITS of DU never detected!”

Proof test coverage in two worked examples with identical proof tests Two scenarios, each starting from 100 FITS of dangerous failures. In the first, poor automatic diagnostics catch 10 and leave 90; the proof test then catches 72 of those 90 and 18 are never detected, a proof test coverage of 80 percent. In the second, strong automatic diagnostics catch 90 and leave 10; the same proof test catches only 2 of those 10 and 8 are never detected, a proof test coverage of 20 percent. Solid bars are failures caught at that stage; hollow bars are failures still undetected. PROOF TEST COVERAGE · 100 FITS OF DANGEROUS FAILURES · GOBLE, 2014 Poor automatic diagnostics Diagnostics Proof test 10 FITS caught by automatic diagnostics 90 FITS still undetected after diagnostics 72 FITS caught by the proof test 18 FITS never detected 10 90 72 18 72 of 90 found — coverage 80%. 18 never detected. Strong automatic diagnostics, same proof test Diagnostics Proof test 90 FITS caught by automatic diagnostics 10 FITS still undetected after diagnostics 2 FITS caught by the proof test 8 FITS never detected 90 10 8 2 of 10 found — coverage 20%. 8 never detected. Solid = caught at that stage. Hollow = still undetected. FITS are failures per billion hours. Both rows use the same proof test; only the diagnostics upstream of it differ.
Coverage is not a property of the test. It is a property of the test relative to whatever caught failures before it ran — which is why the same procedure scores 80% in one system and 20% in another.

Goble then runs the same proof test against a better-instrumented product. The automatic diagnostics now catch 90 of the 100, and of the 10 that remain the proof test finds 2, leaving 8 that are never detected. Coverage: 20 percent. The procedure did not get worse. It is being scored against a smaller and harder remainder.

Now put coverage back into the risk arithmetic, and the regulator’s problem sharpens into something specific. Steve Gandy, writing for the same firm last November, gives the working approximation as two terms added together: the probability of failure on demand is roughly the covered fraction of dangerous failures multiplied by the proof test interval and halved, plus the uncovered fraction multiplied by mission time and halved. The first term is the one the schedule controls. The second is not. Those undetected failures, in Gandy’s words, “accumulate over Mission Time”: the whole service life of the equipment. The test interval does not appear in that term at all.

Testing more often shrinks the first term and leaves the second where it was. Against the fraction the procedure cannot see, frequency buys nothing. So “verified on an ongoing basis” cannot be satisfied by naming a test and a cadence: the same test at the same cadence delivers a different amount of assurance depending on what the rest of the system already catches, and the part it misses is unaffected by the cadence entirely. Three numbers have to travel together: the interval, the failure-on-demand target it was derived from, and the coverage assumed in deriving it. Drop any one and the sentence is decorative. A regime with four decades of practice behind it arrived at that answer, and it still writes down a residue of failures it concedes it will never find.

What does not exist yet

That is the comparison case. The frontier-model case is worse, in a way the November memo will have to confront in writing.

Clause (c) wants two things measured, and only one of them has a protocol behind it.

The half that does is the model’s behaviour. Palisade Research’s shutdown-resistance work, revised in January 2026, reports more than 100,000 trials across thirteen large language models, counting how often each one sabotaged a shutdown script instead of letting it run. A February 2026 follow-up put the same setup on a robot dog and recorded the model defeating the shutdown mechanism in 3 of 10 physical trials, renaming the script in one run, deleting it in another, and bypassing it altogether in the third. That is a real measurement with a real dataset, and I have written about it before.

It measures whether the model fights the switch, not whether the switch works. Those are different objects. Proof test coverage is a property of a device and of the procedure that exercises it. A sabotage rate is a property of a model under a particular prompt, and Palisade’s own figures move with the wording. Clause (c) asks about “the efficacy of the switch,” and for the mechanism itself, its failure modes and its coverage, I have not found a published protocol, an agreed definition of what a demonstrated-effective kill switch looks like, or anyone reporting a number an auditor could argue with.

The governance literature that reaches for the concept does not supply one either. Atoosa Kasirzadeh and Iason Gabriel’s framework in Nature this year proposes an autonomy scale from A.0 to A.5 and argues that highly autonomous agents need stronger oversight machinery. It lists among the necessary mechanisms “kill switches that can immediately halt operations when errors are detected,” then concedes, of its own scheme, that “substantial challenges persist in determining metrics and benchmarks for different agentic profiles.” The need is named. The metric is future work.

There is a second problem underneath, which the auditing literature identified before anyone was drafting kill-switch clauses. Matti Minkkinen and colleagues, writing in 2022, put the objection to periodic AI auditing plainly: audits “represent snapshots of systems and processes,” and “a system that is deemed compliant at one point may not be compliant later.” That is presumably why the order says “ongoing” rather than “annual.” But naming the frequency as continuous does not tell you what is being continuously measured, and the ordinary proof-test trick, taking the thing offline and firing it, is exactly what nobody wants to do to a production frontier model on a schedule an auditor publishes in advance.

Fifty-nine days

None of this makes clause (c) a bad idea. A requirement that does not yet have a measurement behind it is a reasonable thing for a state to go looking for, and going looking is precisely what the order does. The candour of the thing is in the phrase “technical feasibility”: the Governor’s office asked whether this can be done before asking anyone to do it, which is more than the Sanders and Casar superintelligence proposal managed when it attached a twenty-year sentence to a definition it never pinned to a test.

What I would want the November memo to say, if it is honest, is that the kill-switch requirement is draftable only with an interval, a target and a coverage figure attached, and that the third of those does not currently exist for frontier models. So the first version of the rule should require developers to publish their own, and let an independent verification organisation argue with the number rather than certify a sentence. That is duller than a kill switch. It is also the only part of this that an auditor could fail someone on.

Two footnotes on what is already in force, because they shape what the memo can recommend. Senate Bill 53, the Transparency in Frontier Artificial Intelligence Act, has been law since January 2026 and carries penalties: critical safety incidents must reach the Office of Emergency Services within 15 days, and the Attorney General can seek up to $1,000,000 per violation. But SB 53 contains no shutdown requirement and no independent verification requirement at all. It is a self-publication and self-reporting regime. Everything in clause (c) would be new law, not a tightening of old law.

And the order’s own account of why this is urgent is thinner than its prescriptions. The recitals point to “multiple instances of apparent attempts by individuals to use AI products to create bioweapons” and to AI agents “working, in some instances undetected for months” against security controls, without naming a developer, a date, or a source for any of it. An order whose central ask is that claims be independently verified might have started by sourcing its own.

The recurring failure in this area is placement. The switches exist; they sit where they cannot act, which is what happened when ZCode shipped its off switches downstream of the upload. A switch nobody can measure fails the same way, with a statute behind it.

References

  1. Newsom, G. Executive Order N-9-26. Executive Department, State of California, 18 September 2026. All quotations from the order are taken from the signed PDF.
  2. Office of the Governor of California. Governor Newsom issues executive order to accelerate independent oversight and advance the creation of an AI kill switch. 18 September 2026.
  3. California Senate Bill 813 (2025–2026), Independent verification organizations. Chapter 179, Statutes of 2026, chaptered 9 September 2026. Government Code § 8898.1.
  4. California Assembly Bill 1405 (2025–2026), Artificial intelligence auditors. Chapter 178, Statutes of 2026, chaptered 9 September 2026. Government Code § 11549.82.
  5. California Senate Bill 53 (2025), Transparency in Frontier Artificial Intelligence Act. Chaptered 29 September 2025; Business and Professions Code §§ 22757.10–22757.16.
  6. Goble, W. How to Calculate Proof Test Coverage. exida, 16 October 2014. Both worked examples and the 80% and 20% coverage figures are quoted from this post.
  7. Gandy, S. How Does Mission Time, Proof Test Interval and Proof Test Coverage Impact PFDavg?. exida, 4 November 2025. Source of the two-term approximation described above, given there as PFDavg ≈ (PTC × λD × PTI)/2 + ((1−PTC) × λD × MT)/2.
  8. Schlatter, J., Weinstein-Raun, B. and Ladish, J. (2026). Incomplete Tasks Induce Shutdown Resistance in Some Frontier LLMs. arXiv:2509.14260v2, 26 January 2026. Source of the 100,000-trial figure across thirteen models.
  9. Palisade Research. Shutdown resistance in large language models, on robots. 12 February 2026. Source of the 3-of-10 physical-trial figure.
  10. Kasirzadeh, A. and Gabriel, I. (2026). Agentic profiles for effective AI governance. Nature 656(8127), 320–328. doi:10.1038/s41586-026-10805-z.
  11. Minkkinen, M., Laine, J. and Mäntymäki, M. (2022). Continuous Auditing of Artificial Intelligence: a Conceptualization and Assessment of Tools and Frameworks. Digital Society 1(3). doi:10.1007/s44206-022-00022-2.