Who, If Anyone, Should Have a License to Kill AI?

Highly capable artificial intelligence (AI) systems have sparked concerns about catastrophic outcomes, including the potential end of humanity, among lawmakers, lab representatives, and the American public. After a series of “rogue” AI incidents this summer, six in 10 U.S. adults consider the possibility that AI will “someday destroy humanity” to be at least a moderate risk, and half say development should be paused. 

Fearing the possibility of existential risk, both federal and state officials have proposed legislation to mandate the implementation of kill switches at leading AI companies. In a traditional factory, such a switch would perhaps involve a lever with the power to shut off all the machines in one fell swoop. For AI systems, however, it’s trickier to describe what exactly a kill switch would entail, who would hold the authority to trigger it, and how it would be enforced in practice. The recent surge in interest in such a far-reaching power invites serious scrutiny of these questions. 

Who Triggers the Switch?

Policymakers across party lines view kill switches as a timely intervention to address the scale and immediacy of AI threats. However, proposals so far lack consensus on who should hold the authority to turn the systems on or off.

At one end of the spectrum, Sen. John Kennedy (R-La.) proposes leaving the decision for an emergency shutoff with the AI companies themselves, rather than a public authority. He argued that a “fire extinguisher” is needed for recursive self-improvement, a technique U.S. frontier labs are pursuing to use AI to train other AIs, which Kennedy contends could turn the technology into “an independent species” with its own goals. Despite its ambition, the bill has gone nowhere so far: In September, Sen. Rand Paul (R-Ky.) blocked Kennedy’s attempt to pass a kill switch by unanimous consent.

Other proposals involve public authorities, at least in a verifying capacity. In California, Gov. Gavin Newsom tasked the state’s Government Operations Agency and the Office of Emergency Services to convene national experts and “advance the creation of a ‘kill switch’ for frontier models,” with an independent body assigned to verify the efficacy of labs’ respective switches on an ongoing basis.

In the House, a bipartisan group led by Reps. Ted Lieu (D-Calif.) and Nathaniel Moran (R-Texas) introduced the AI Kill Switch Act, which would permit the Department of Homeland Security to verify that qualifying AI labs can halt inference, block user access, suspend access for specific use cases, and “shut down” systems entirely. The bill also directs the department to consider a “graduated deployment-corrections framework,” which would weigh lesser measures against the “severity and immediacy” of a given risk. 

The bill goes farthest among current proposals in specifying the events that would trigger such a shutdown. These include an AI model trying to conceal its capabilities or evade shutdown orders, incidents that cause at least 10 deaths, economic damages of at least $100 million, or other loss-of-control scenarios—defined as “a scenario in which a covered technology pursues outside of red-teaming or other structured testing a goal that is not a goal intended by the developer or operator of such technology.”

Companies that fail to comply with these demands face severe penalties, including up to $2 million per day for lacking shutdown capability in their AI systems, and up to $20 million per day for defying a Department of Homeland Security shutdown order. 

Some of the scaffolding for these proposals already exists in New York’s Responsible AI Safety and Education (RAISE) Act. Introduced in March 2025 by Assemblymember Alex Bores and State Senator Andrew Gounardes and signed into law by Gov. Kathy Hochul, the law requires major frontier AI developers to report critical safety incidents—including technical control failures or autonomous actions beyond user intent—to the DIGIT Office within 72 hours of determining an incident has occurred. Incidents presenting an imminent risk of death or serious physical injury require a separate disclosure to authorities within 24 hours.

The law will soon go into effect: Frontier developers such as OpenAI and Anthropic must register with the state starting in November, with full compliance and enforcement beginning on Jan. 1, 2027. The RAISE Act confers no shutdown power itself. It does, however, offer an accountability pipeline that future regimes could rely on.

Finally, there’s a third possibility for a triggerman: independent third parties. The New York City Council, whose proposed bill defines shutdown capability as the ability of “a human operator to cause a model to stop functioning temporarily or permanently,” assigns an outside third party to verify whether each lab has an effective switch. 

The sheer number of kill switch proposals indicates real demand for more control over frontier systems. What remains unclear is whether these proposals are technically feasible, and what circumstances would justify pulling the trigger.

What Would It Switch Off?

The labs themselves say shutdown is possible. Jack Clark, co-founder of Anthropic, told the BBC that Anthropic has the capacity to do so and asserted that “most labs have different ways of being able to pull the plug.” Just because the labs can, however, doesn’t mean they will. Clark acknowledged as much, suggesting that a means of shutting off AI software completely was something governments “might want to eventually pass rules around.”

Researchers at cybersecurity firms and leading universities are more skeptical about what such rules would achieve in practice. The barriers to shutting down an AI model or feature are the result of the complexities baked into the algorithmic ecosystem. Unlike a machine running on a factory floor, or a state-controlled nuclear reactor, AI systems are not geographically concentrated. They exist in a sprawling and interconnected digital world, drawing inputs from countless sources and delivering outputs to billions of devices.

Data centers, run by companies such as Amazon, Alphabet, and Meta, are scattered across the world and comprise thousands of machines, chips, servers, and backup systems. A kill switch would therefore have to reach not only the primary systems but also the backups built specifically to save workloads during outages.

Here emerges a paradox. As governments, critical infrastructure providers, and private firms become increasingly reliant on highly capable systems, they are demanding assurances that services will not be interrupted. Labs have responded by making their systems more resilient—precisely the property that a kill switch must defeat. Moreover, because AI systems perform so many different functions within a given organization or company, a single switch is unlikely to suffice; businesses would require several switches, each tailored to a particular task.

The scope of AI deployment across sensitive functions compounds these challenges. Whoever orders the switch must avoid shutting down more functions than necessary, and an overbearing shutdown risks serious disruption across critical sectors. A hospital might rely on a frontier model to flag patient conditions or triage emergency admissions. If that model were suddenly switched off because of an unrelated incident—for example, an agent hacking into a financial firm’s database—the hospital could risk losing its functions with little or no warning.

Even those most worried about AI have questioned whether a switch would settle the matter. Jacob Coxon, the researcher who left Anthropic over fears that companies are racing to build “out-of-control” AI, has suggested that kill switches might help in the short term but cannot guarantee long-term safety. Instead, he argued that tech companies should slow the development of frontier models—though this, again, raises questions about who would have the authority to do so, what such a slowdown would entail, and how long it would be enforced.

Risks, Old and New 

Open-weight models further complicate the scope question. Once downloaded, users can run open models on any computer around the world, at significantly lower cost, with no tie back to the original developer. That independence makes a universal shutdown nearly impossible to enforce. Notably, Lieu has said the House bill would not cover open-weight models, though he did not rule out adding them to the alert system later.

This caveat is important, because a mandate requiring developers to retain control over every copy of their models could effectively foreclose open releases, forcing American consumers to rely on closed-weight systems. Such a requirement would monopolize even more power in a handful of frontier firms that maintain sole control over their “black box” models. It would also stifle long-term innovation, forcing researchers and smaller startups to use expensive, proprietary application programming interfaces (APIs) or foreign—likely Chinese—alternatives.

Lawmakers should therefore state plainly whether their proposals require control over a provider’s own deployments or continuing control over models already released to the public. The latter would determine who may develop, study, and use AI without a provider’s ongoing permission.

Even if these technical and jurisdictional questions can be resolved, the switch itself could introduce new risks. Luke O’Grady of the Center for Cybersecurity Policy & Law compares kill switch requirements to the Clipper Chip, which the National Security Agency proposed in the 1990s to encrypt voice and data messages with a hidden interface that would allow federal, state, and local law enforcement to decode intercepted information. Both involve deliberately engineering a technical backdoor that bad actors could exploit, manifesting, for example, as threats to user privacy.

As more people hand their inboxes, calendars, and credit cards to personal agents such as Meta’s Muse, bad actors’ incentive to compromise those systems will only grow. A built-in kill switch would be the icing on the cake. There is also the risk of extortion, because individuals or entities that seize control of a switch could threaten every hospital, business, or individual dependent on the system and coerce them into paying vast sums of money. In other words, a mechanism designed to contain risk could become a risk in itself.

All of this highlights why something that may seem like a reasonable request is far more technologically challenging and could have far-reaching consequences for national security and technological innovation.

What Justifies the Switch?

Assuming switches can be developed and codified into law, the trickiest question may be which conditions would justify such an extreme intervention. The Lieu-Moran thresholds specify circumstances under which the trigger would be justified, but “other loss-of-control scenarios” leaves wide discretion, and none of the current proposals clearly define what entails a “frontier” model in the first place. 

This raises crucial questions. Would a switch immediately kill all the models of a given lab, or only the generation responsible for the offending behavior? Would it cross jurisdictions—for example, could a U.S. American lab, or a U.S. regulator, switch off a model integrated into a British company’s operations? And what would justify pulling the trigger? If a swarm of OpenAI agents breached a company such as Hugging Face, as happened in July, would that suffice? By now, there have been several incidents across multiple labs of agentic AIs “behaving contrary to the instruction of [their] developer or operator,” a condition that would, in theory, justify killing the technology under the Lieu and Moran bill.

It’s worth noting that Washington has already improvised an AI shutdown. In June, the Commerce Department used its export control authority to pull two Anthropic models offline—Mythos 5 and Fable 5—allegedly over national security concerns about their cyber capabilities. In the absence of a kill switch, the government reached for trade law instead. The controls were lifted on June 30 and access was restored the following day, but the episode revealed how easily a power to shut down AI systems can be used—and potentially abused—without a dedicated legal authority or a set procedure to restore access.

Rather than imposing a unilateral shutdown mechanism, several less drastic tools exist that could better address the risks behind these proposals. O’Grady contends that Congress should mandate more rigorous internal testing, red-teaming exercises that expose vulnerabilities during earlier stages of development, and more precise liability regimes to identify who would be held liable for qualifying incidents. Dylan Baker of the Distributed AI Research Institute similarly argues that the switch metaphor obscures the more basic need for concrete safety measures.

Timing, as ever, is of the essence. The government, they emphasize, should act on a model before it causes harm, rather than after the fact.

What’s the Alternative?

Lawmakers should be able to specify the precise conditions under which a kill switch would be enforced and how a switch would operate in practice within a hyperconnected digital world. They must also address the risk that extending shutdown mandates to open models could entrench a monopoly among large, closed-weight providers. Such a shift would favor one particular method of development and restrict the public’s ability to build and use these tools independently. If the answers to these questions remain elusive, a kill switch should not be implemented. 

There are at least three alternative paths to address the concerns underlying kill switch proposals that stand on more defensible legal ground and raise fewer technical questions. Crucially, these paths are not mutually exclusive and should be pursued in tandem.

The first is to mandate greater transparency from labs about their model capabilities and the safeguards in place against misaligned behavior. SB 53, now law in California, imposes related constraints on large frontier AI labs but includes easy workarounds, such as limiting what mitigations a lab formally shares with the state—and is thereby held accountable for—which should be corrected at the federal level.

The second is to establish a federal research effort that draws on lab, academia, and civil society expertise to develop and share narrowly scoped, effective mitigations that can then inform standards and regulation. This multistakeholder approach has precedent in cybersecurity (the National Institute of Standards and Technology’s National Cybersecurity Center of Excellence), aviation (the Commercial Aviation Safety Team), and SEMATECH, a research consortium of the Defense Advanced Research Projects Agency and chipmakers developed in the 1980s. All these structures combined federal funding with industry expertise to develop solutions for specific security problems, rather than attempting a one-size-fits-all approach.

The third is to reinforce the mechanisms described above—improved internal testing, earlier and more rigorous red-teaming, and clearer liability rules for qualifying incidents—that would reduce the odds of labs deploying models that require a kill switch in the first place. 

One thing is certain: The AI labs cannot continue chasing the frontier full-speed ahead without installing breaks that provide the assurances needed to increase adoption and reduce the odds of loss-of-control scenarios. But these brakes should not necessarily take the form of an on-off switch that can shut down the technology altogether—a proposal that may sound good in a press release but is highly improbable and risky in practice.