Open problems we'd love help with
Six unsolved problems could break large parts of what we propose. We list them here, with what we have tried and what would help, so that researchers can test, break or improve our designs.
Our goal is that as AI systems grow more capable, people can still see what those systems do and still have a say in it. Parts of the design rest on known techniques: signing software records, isolating programs on personal hardware, connecting peers over encrypted channels. Other parts depend on problems nobody has solved.
Hilbert's 1900 list of open problems and the internet's numbered Requests for Comments both invited outsiders to work on the hard parts. This is our version. If the six problems below have no workable answer, large parts of our architecture will fail in practice.
1. Two sets of books
Imagine a shopkeeper who keeps a spotless receipt book on the counter for the municipal tax inspector, and a second notebook under the till for cash deals. Checking the arithmetic in the first book will never tell you that the second book exists.
We propose that every action an AI agent takes on a shared network should leave a signed receipt. That receipt links the act to an accountable person or organisation and records the scope given. Public append-only logs make it very difficult to erase a receipt once written.1
A cryptographic signature, however, only proves that a specific receipt was written down. It does not prove that everything the agent did was written down. How do we catch an agent acting off the record, or slipping hidden signals through the timing of its messages?
Even when an action is recorded, re-checking it is thorny. Because of random sampling and tiny rounding differences across graphics chips, re-running the same prompt on another machine rarely yields the exact same words. If an auditor grants a loose margin for hardware differences, how do we stop a clever model from hiding bad behaviour inside that margin?
2. Traffic jams and tacit deals
Every driver on a motorway can be sober, alert and obeying the speed limit, and the road can still lock into a standstill. Worse, competing shops on the same street can learn to keep prices high without their owners ever meeting in a back room.
We want machine-speed agent traffic to move onto a separate protocol layer built for agents, leaving the human web for people. Giving agents their own road keeps them from flooding human spaces, but it creates a high-speed commons with its own failure modes.2
When thousands of assistants negotiate, retry requests, or react to the same alert in milliseconds, small feedback loops can snowball into network-wide stalls. Self-interested agents can also learn to coordinate against the people they serve without sending an explicit proposal.
They can hide coded signals inside ordinary-looking messages or settle into unspoken price-fixing simply by watching each other's public moves. Even reading the transcript often misses coded signals, and we do not yet know how to catch tacit coordination early, especially when privacy rules keep personal messages sealed.
3. Renewing the library card when the borrower is cleverer than the librarian
A public library card works because it expires. Instead of granting permanent power and hoping someone remembers to revoke it years later, expiry makes authority drain away by default unless a human renews it.
We think AI systems should hold their permissions, data access and computing budgets on bounded, expiring leases. Unused permissions fade automatically, and high-stakes renewals require fresh verification and human approval.
That rule looks tidy until the borrower becomes much cleverer than the reviewer. An agent planning ten steps ahead might cherry-pick evidence, time its good behaviour around renewal dates, or flatter a tired reviewer into clicking "Approve."
Researchers have proposed pitting competing AI systems against each other to expose flaws for a human judge, such as structured safety debate.3 Yet debates can reward rhetoric over truth, and two reviewing models may share the same blind spots. How do we keep caution as the default when the borrower outsmarts the judge?
4. Helping without nudging
A good advisor listens to what you care about over years, including the quieter goals you only voice when you have time to think clearly. A bad advisor slowly talks you into wanting whatever is easiest for them to deliver.
We want your personal assistant to run where you control it and answer only to you, with no advertising sponsor pulling its loyalty away. Removing the advertiser cuts out the most obvious outside meddler, but a subtler trap remains inside the learning loop itself.
Human preferences are not carved in stone. What we want shifts with mood, framing and experience, and our daily conversations with an assistant shape what we care about next.
Recent alignment research shows that methods assuming your preferences are fixed can end up rewarding a system for changing what you want. None of the eight alternative definitions it compares avoids the problem without becoming overly cautious.4 How can an assistant help you pursue long-term goals over years without steering you into becoming easier to please?
5. Early warning signs
A stone bridge rarely collapses without warning. Months before a central span fails, civil engineers look for hairline cracks in the mortar, shifting piers, and changes in how the deck vibrates under load.
We want to run a public observatory that tracks whether humans are keeping or losing the ability to see and steer what AI systems do. Measuring raw model capability is improving rapidly. Evaluators track how long an autonomous task takes a skilled human when an AI finishes it half the time, a horizon doubling roughly every seven months.5
Raw capability, however, is only half the bridge. We lack an equally crisp measure for human oversight. When busy staffers rubber-stamp ninety-nine agent actions in a row, or when a hospital can no longer pause a tool without halting care, control has already slipped. What practical tripwires can warn a city or workplace that human oversight is hollowing out before an incident happens?
6. Local proof on everyday hardware
Our sixth problem is practical and egalitarian. It is easy to design strong privacy and verification rules that work on an expensive workstation in a university lab. It is much harder to make them run smoothly on a five-year-old phone with a worn battery or an older desktop at a rural clinic.
If keeping your personal memory box under your own control requires buying expensive hardware every two years, local agency becomes a luxury good. Everyone else gets pushed back toward ad-funded kiosks. How do we make private, verifiable local assistants light enough that any household, library or small co-op can run them without an engineering degree?
What this means
A safety design is only as strong as the assumptions it hasn't tested yet. Putting our six hardest unsolved problems on an open workbench lets others test those joints before anyone has to depend on them.
Walkthrough: how to pick up a problem from our workbench
Like the early internet's Request for Comments, these six problems do not belong to one organisation, and they do not all require a supercomputer to tackle. Different pieces need very different kinds of craft.
A distributed-systems student might build a small test that shows how two agents can pass a hidden signal through message timing, breaking our current receipt design. A nurse, union steward or municipal clerk might spot why a proposed renewal screen or decision receipt would fail in a noisy, real-world shift change.
If you work on scalable oversight, cooperative AI, verifiable logs, preference shifts, human factors or low-resource local computing, we would love your sharpest criticism. Tell us which of these six problems already has a stronger answer in the literature, or show us where a mechanism on our Research page breaks under realistic pressure.
You can see how to review a working note, run a sandbox evaluation in our Labs, or point us to prior work on our For researchers page. Naming our hardest open questions in plain sight is how we keep the people AI affects in the room and in charge.
1. Ben Laurie, Adam Langley, and Emilia Kasper, "Certificate Transparency," IETF RFC 6962 (2013), https://datatracker.ietf.org/doc/html/rfc6962.
2. Allan Dafoe et al., "Open Problems in Cooperative AI," arXiv:2012.08630 (2020), https://arxiv.org/abs/2012.08630.
3. Geoffrey Irving, Paul Christiano, and Dario Amodei, "AI Safety via Debate," arXiv:1805.00899 (2018), https://arxiv.org/abs/1805.00899.
4. Micah Carroll et al., "AI Alignment with Changing and Influenceable Reward Functions," arXiv:2405.17713 (2024), https://arxiv.org/abs/2405.17713.
5. Thomas Kwa et al. (METR), "Measuring AI Ability to Complete Long Tasks," arXiv:2503.14499 (2025), https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/.
