No one should command a swarm

Large undertakings used to need many people to carry them out, and each of them could question an order, warn a colleague or refuse. That friction was messy, and it stopped many disasters. Software agents remove it. One person with one credential can now instruct thousands of agents that don't hesitate or ask whether an instruction makes sense.

Engineers already match controls to consequences: a garden tap turns with one hand, while a dam's floodgates need separate keepers who check the river first. Systems that let agents act at scale need the same graded checks.

Two keys on the vault door, and a fuse box in the hall

We already know how to handle single points of failure in physical life. A bank vault requires two keys held by two different officers. Commercial aviation checklists require the pilot flying and the co-pilot monitoring to cross-check critical switches aloud before either hand moves.¹

Everyday engineering also separates stopping power from starting power. Think about the electrical fuse box in your hallway. When a defective kitchen toaster shorts out, the local breaker trips in a few milliseconds to stop a house fire.

Nobody has to call a committee to cut electricity to a smoking toaster. Stopping local harm is instant and automatic. By contrast, energizing a high-voltage substation that feeds fifty thousand homes requires multiple grid engineers, physical lock-out tags, and staged safety checks.

Computer security reached the same conclusion fifty years ago. In 1975, Jerome Saltzer and Michael Schroeder called it separation of privilege: high-stakes actions should require two independent keys.² Four years later, Adi Shamir showed how to split a mathematical key into shares so a chosen fraction of guardians must agree.³

Today, separation of duties, dual authorization, and threshold cryptography are standard rules in government systems and critical infrastructure.^4,5 Yet the way we deploy AI agent fleets routinely ignores all three.

One assistant per person

Think about the difference between a personal assistant and an outward-acting swarm. An assistant that works only for you (sorting mail, checking a medical bill, or helping your teenager practise algebra) keeps power close to home.

If your personal assistant misunderstands a request, the mistake stays inside your own study. You notice the odd draft on your screen, tap the stop button, and fix the instruction.

The danger begins when one account can dispatch a fleet of agents into the shared world. Three things go wrong at once:

• One mistake scales instantly. A typo, a misread alert, or a moment of panic can trigger thousands of outward actions across markets and offices before anyone can read a warning banner.
• One stolen key hands over the fleet. An attacker does not need to defeat a whole institution if all outward authority sits behind one administrator credential.
• Human oversight becomes a polite fiction. Claiming one person is "in the loop" over ten thousand fast-moving agents is like asking one lifeguard to watch ten thousand swimmers in rough surf at night.

Walkthrough: power grids and clinic supplies

To see why this distinction matters, consider two ordinary community systems where automated agents are beginning to coordinate work.

First, picture a regional electric utility where forty thousand homes have rooftop solar panels and garage batteries. Each homeowner runs a personal agent to charge their battery when power is cheap and sell extra electricity back when neighbours need it.

Suppose a vendor builds a central console that can send one override command to all forty thousand home batteries at once. One afternoon, a faulty weather forecast or a hijacked laptop instructs every battery in the valley to dump full power into the street lines simultaneously. Within seconds, local transformers overload and trip across three towns.

Now picture the same valley governed by fuse boxes and two-key rules. Each household battery has a local hardware breaker that answers only to the homeowner and physics. If an external signal asks for an unsafe surge, the local fuse trips in milliseconds.

Meanwhile, at the utility level, no single console can command more than one feeder line. Any adjustment across multiple substations requires independent digital sign-off from two senior grid operators, an automated physics check, and a ten-minute ramp.

Second, consider a cooperative of rural clinics using agents to order seasonal vaccines and surgical supplies. If one central account can command two hundred clinic agents to place urgent bulk orders at the same second, a single garbled prompt can drain regional warehouses and leave neighbouring hospitals empty-handed.

With risk-scaled sign-off, a clinic nurse can still let their local assistant reorder a routine box of gauze with one click. However, any coordinated bulk order above normal monthly volume pauses at a two-key gate until both the clinical pharmacist and a regional supply steward turn their keys.

Matching agreement to the size of the risk

Instead of hoping that a single commander never errs, we think the rules should govern how instructions travel across four clear rungs.

First, one person directs one personal agent. You can run as many calculations as you like inside a closed sandbox that touches no one else. Inside your own home, your say-so is enough.

Second, stopping local harm never waits for a committee. Like the fuse box in your hallway, any person or local operator can trip a defensive circuit breaker immediately to pause an erratic agent or close a leaking port.

Third, actions that affect a group require two-person control. When an automated workflow modifies shared workplace records or contacts hundreds of residents, the person who drafts the instruction and the person who approves its release must be two distinct individuals.^1,4

Fourth, city-scale actions require threshold sign-off and a cooling-off clock. Changes touching public utilities or regional markets should require cryptographic shares from independent operators and accountable public representatives, paired with a public waiting window.^3,5

Where this is hard

These guardrails are hard to build. Four problems stand in the way, and we are testing them in simulation rather than claiming they are solved.

First, how do you verify that approvals come from distinct humans rather than fake accounts, without building an invasive biometric surveillance database? Second, how do you stop a clever agent from slicing one big dangerous order into a thousand tiny requests that each slip just under the two-key threshold?

Third, who estimates the risk of an action when the agent proposing it has every incentive to call it routine? The gatekeeper that scores the blast radius must be independent of the model that drafts the plan, relying on plain, auditable rules rather than persuasive prose.

Fourth, how do we prevent alarm fatigue so reviewers do not rubber-stamp high-stakes prompts out of habit?¹ In cockpits and hospital wards, when warning chimes sound every two minutes, tired operators eventually click approve without reading. Two-key gates only stay sharp when everyday local chores run on their own, reserving shared sign-off for moments that genuinely carry community weight.

Why builders and officials both gain

Good guardrails protect the people in charge as much as the public. When bank robbers know a branch manager cannot open a two-key time-locked vault alone, threatening that manager stops working.

The same logic protects software engineers and public officials. When no single official holds a master key that commands a swarm alone, no one can be blackmailed, tricked by a deepfake call, or pressured at midnight into turning it alone.

We put two locks on watergates and bank vaults, checklists in cockpits, and fuse boxes in our hallways because we respect how much is at stake. Bringing that same calm discipline to AI lets your household enjoy a helpful personal assistant, knowing no stranger at a distant console can unleash a fleet that upends your community.

1. Federal Aviation Administration, Crew Resource Management Training, Advisory Circular AC 120-51E, U.S. Department of Transportation, January 2004 (cross-verification, challenge-and-response checklists, and error trapping in multi-crew cockpits). faa.gov/documentLibrary/media/Advisory_Circular/AC_120-51E.pdf

2. Jerome H. Saltzer and Michael D. Schroeder, "The Protection of Information in Computer Systems," Proceedings of the IEEE, vol. 63, no. 9, pp. 1278–1308, September 1975 (Section I-A3, "Separation of privilege"). doi.org/10.1109/PROC.1975.9939

3. Adi Shamir, "How to Share a Secret," Communications of the ACM, vol. 22, no. 11, pp. 612–613, November 1979. doi.org/10.1145/359168.359176

4. National Institute of Standards and Technology, Security and Privacy Controls for Information Systems and Organizations, NIST Special Publication 800-53, Revision 5, September 2020 (Control AC-5, Separation of Duties, and Control AC-3(2), Dual Authorization / Two-Person Control). doi.org/10.6028/NIST.SP.800-53r5

5. Luís T. A. N. Brandão, Nicky Mouha, and Apostol Vassilev, Threshold Schemes for Cryptographic Primitives: Challenges and Opportunities in Standardization and Validation of Threshold Cryptography, NIST Internal Report 8214, National Institute of Standards and Technology, March 2019. doi.org/10.6028/NIST.IR.8214