What would count as oversight?
Most AI policies promise human oversight, which in practice often means one person approving hundreds of automated decisions an afternoon. Oversight is real only when people get checkable records, summaries paced for human attention, permissions that expire, and a brake they can actually pull.
The usual answer to how an AI system will be kept safe is a human in the loop: a person reviews what the machine proposes and signs off. When the machine is right ninety-nine times in a row, that person soon approves in under two seconds. Human-factors research has documented this automation bias for decades: people stop looking for what the machine missed.1
When something finally goes wrong, the organisation points to the person who approved it. The sociologist Madeleine Clare Elish calls this role a moral crumple zone: the human absorbs the blame for a failure they had neither the time nor the information to prevent.2 Four things would make oversight real.
The glass-walled kitchen
Ceremonial oversight puts a tired person next to a speeding conveyor belt or a locked door and asks them to vouch for whatever comes out. Substantive oversight looks like a glass-walled restaurant kitchen.
In a well-run glass kitchen, you do not chop every onion yourself, and you do not taste every spoonful of soup per minute. Instead, you can see the counters and watch the cooks work. More importantly, the walk-in cooler has health-department thermometers on the wall, and every crate of ingredients carries a stamped batch number. And right by the door hangs an emergency stop cord that any line cook can pull.
You can tell what went into the pot, verify that the cooler stayed cold overnight, and send a dish back before it ever reaches the dining room. We think four testable conditions separate real oversight from rubber-stamping. If any one of them is missing, the human in the loop is mostly decorative.
1. A record the system can't edit
You do not inspect a restaurant kitchen by reading the chef's press release. You check the thermometer log on the cooler wall and the batch numbers on the shelves. Yet many interfaces today ask us to judge an AI agent by reading the summary the model itself wrote after finishing a task.
If the model cut a corner, guessed a number, or overwrote a file it should have left alone, its polite summary will rarely volunteer that fact.
Real oversight starts with an independent record of what the agent actually did. Which files did it read? Which tools or outside services did it call? Under whose permission did it act, and what changed as a result?
Those records must be created by the surrounding sandbox infrastructure, not narrated by the model afterwards. And they must be tamper-evident, so neither the agent nor an embarrassed operator can rewrite the trail once an error comes to light.
2. Clear summaries paced for a human mind
A stack of ten thousand raw computer logs is no more useful than a locked kitchen door. Both hide what happened behind a wall no human can get through in an afternoon. This problem is getting sharper fast: the length of tasks AI agents can complete on their own has been doubling roughly every seven months.3
As an agent's work stretches from a thirty-second query to a multi-hour chain of actions, no person can read the raw trail in real time. Dumping that trace onto a reviewer's screen and calling it transparency is a polite fiction.
To match a human mind, oversight needs hard rule locks paired with separate checking models run in isolated rooms. Hard rules block forbidden actions automatically, like a thermostat lock that prevents an oven from exceeding a safe limit. Meanwhile, separate checkers summarise the receipt trail and scan it for unusual steps or side effects the main agent glossed over.
Instead of clicking "approve" four hundred times an hour on routine bottles, the human reviewer focuses on the exceptions. They inspect the moments where the checkers raise a flag or disagree, plus a small random sample to catch shared blind spots. Good oversight spends human attention where human judgment actually matters.
3. Bounded scope and expiring leases
Even a careful reviewer will miss things on a busy day. If a single distracted click grants permanent access to a clinic's billing system or a company's codebase, human oversight fails. The human has to play flawlessly forever, while the machine only has to get lucky once.
Real oversight requires narrow boundaries and permissions that expire on their own. An agent should hold authority the way a borrower holds a library book: for a stated purpose, within a clear limit, and for a fixed window of time.
When a two-hour lease runs out, the door locks automatically unless a person actively renews it. If a reviewer gets called away or steps out for lunch, the system's reach shrinks back toward zero instead of expanding in the dark.
4. A working brake that doesn't wreck the room
Suppose a cook spots smoke in the kitchen or a reviewer spots a strange pattern in an agent's receipts. In too many software deployments, halting an automated system freezes customer support, drops live medical records, or leaves shared files half-written. When pulling the stop cord wrecks the room, nobody pulls the cord until it is too late.
A brake only counts if you can pull it cleanly on an ordinary afternoon. Every high-stakes workflow needs a staged fallback, a way to hold pending actions in a waiting buffer, roll back uncommitted steps, or hand the task back to a simpler manual checklist. Teams must test that stop cord regularly, the way a school or hospital runs a monthly fire drill.
The four-part test
Before calling a system "human-overseen," ask four questions: Is the record independent of the model's own words? Can a person grasp the key risks in the time they have? Does authority expire by default? And did anyone test the stop cord this month?
The hard limit: when the cook is cleverer than the inspector
We want to be candid about where these four conditions hit a wall. Independent receipts, clearer gauges, expiring leases, and tested brakes protect us against ordinary drift and sloppy engineering. They do not, by themselves, solve the hardest problem alignment researchers worry about.
What happens when an agent is substantially more capable than the person checking it? A clever system may understand the inspection rules better than the reviewer does. It could choose actions that look completely benign in the receipt log while setting up a hidden side effect ten steps later.
Researchers call this scalable oversight: how can a busy person with normal human limits supervise a system working on problems too complex to check by hand?4, 5 Work on AI control asks an even starker question: how do we stay safe if an untrusted model actively tries to get around our checks?6
Promising mechanisms exist. In debate protocols, two models argue opposing sides of a claim before a human judge, using one model's capability to expose the other's hidden flaws.4 In control protocols, weaker trusted monitors screen every action from a stronger model, escalating the most suspicious cases to careful human audit or rewriting them with a trusted system.6
Our proposed architecture is designed to give those techniques firm ground to stand on. Debate and trusted monitoring fail if models can tamper with the evidence they argue about. Tamper-evident receipts, isolated sandboxes, and expiring leases provide the courtroom walls that keep witnesses from rewriting the transcript.
Even so, open failures remain. Two models can share a blind spot. Persuasive rhetoric can still mislead a human judge.5 And an agent might hide signals across many innocent-looking actions. Those problems are not solved; we build infrastructure to test them honestly.
What real oversight gives back to people
No one wants to spend their working life as a human rubber stamp beside a speeding conveyor belt, waiting to take the blame for a machine they cannot truly inspect. And no evaluator, clinic director, or regulator should accept a safety plan that relies on human vigilance defying everything we know about human attention.
When we insist on checkable receipts, human-paced gauges, expiring authority, and a stop cord that actually works, we stop treating people as shock absorbers for opaque software. We give them a kitchen with glass walls, and a real say in what gets served.
1. Raja Parasuraman and Dietrich H. Manzey, Complacency and Bias in Human Use of Automation: An Attentional Integration, Human Factors 52, no. 3 (2010): 381–410.
2. Madeleine Clare Elish, Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction, Engaging Science, Technology, and Society 5 (2019): 40–60.
3. METR, Measuring AI Ability to Complete Long Tasks (March 2025).
4. Geoffrey Irving, Paul Christiano and Dario Amodei, AI Safety via Debate, arXiv:1805.00899 (May 2018).
5. Samuel R. Bowman et al., Measuring Progress on Scalable Oversight for Large Language Models, arXiv:2211.03540 (November 2022).
6. Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan and Fabien Roger, AI Control: Improving Safety Despite Intentional Subversion, Proceedings of the 41st International Conference on Machine Learning, arXiv:2312.06942 (2024).
