Why a public decision cannot hide in a black box
When software makes official decisions, the public is owed the same reasons it is owed for any exercise of public power. Agencies using AI should keep an open record of each model and how it is used, re-review the system whenever the model changes materially, and allow independent audits.
A planning board can show anyone the calculations behind a bridge. Administrative law has long required agencies to keep records and give reasons, so courts and the public can check their decisions. Software that decides who gets benefits or permits often comes with neither, because the model is proprietary or its reasoning can't be inspected.
Public power requires public reasons
Governments carry a unique responsibility. When a private company buys automated software, it answers to its shareholders. If a shopping app makes a clumsy product recommendation, you can simply close the browser.
Public administration is different. When an agency decides who qualifies for disability support, who receives emergency housing, or whose tax return faces an audit, it exercises the power of the state. Those choices change people's lives in deep, lasting ways.
In a democracy, official power comes with a fundamental bargain. Public power requires public reasons. If an agency denies your permit or cancels your benefits, you have the right to know why.
You have the right to see the rules that were applied. You have the right to check if the facts were accurate. And you have the right to appeal if someone made a mistake.
Proprietary software breaks that democratic bargain. When agencies buy closed algorithms from private vendors, the vendor often claims that the software's inner workings are confidential trade secrets. When a citizen asks why their assistance was cut, the agency staff can only shrug and point at the computer screen.
Three pillars of the scrutable state
We do not have to accept uninspectable governance. Legal scholars and administrative researchers are formulating a clear standard known as the scrutable state.1 It replaces blind trust with three concrete structural safeguards.
First, public agencies must mandate open model dossiers. Before any automated system touches a public case, the vendor must deliver a complete, open record of how the tool was tested, where it fails, and what boundaries govern its outputs. The public body, not the private vendor, must own this dossier.
Second, procurement rules must establish material-change triggers. Software running in the cloud rarely stays still. When a vendor updates model weights, changes system instructions, or swaps out data pipelines, the tool is no longer the system the agency approved. A material-change trigger automatically pauses automated decisions until independent testers re-verify the new behavior.
Third, administrative review must enforce deference to audit. Courts have long given administrative bodies leeway when they make technical calls. But that trust must be earned. Under this standard, courts only presume a process was fair if the agency presents an unbroken, independent audit trail of the actual decision.
Canada's Directive on Automated Decision-Making demonstrates this principle in practice.2 It requires federal departments to publish algorithmic impact assessments, maintain human oversight, and provide plain-language explanations to affected individuals.
Similarly, the European Union's Artificial Intelligence Act classifies public benefit allocation systems as high-risk technologies.3 The statute mandates strict technical documentation, continuous logging, and independent pre-deployment testing. In the United States, federal guidance directs agencies to waive vendor trade-secret claims when algorithms affect citizen rights.4
Inspection builds lasting public trust
Inspecting software does not mean every citizen must read raw code. Very few people calculate concrete load ratios before walking across a municipal footbridge. We cross with confidence because independent inspectors verified the math before the bridge opened to traffic.
Public software deserves that exact same standard of care. When agencies insist on open dossiers and verifiable change triggers, they protect both the public and themselves. They ensure that civil servants remain the masters of their tools rather than captive customers of opaque algorithms.
The choice before public institutions is not between adopting technology or rejecting it. The choice is whether our public tools serve democratic accountability or chip away at it. By insisting on inspectable blueprints, communities ensure that the power of the state always remains accountable to the people it serves.
Explore actionable guidance on establishing model dossiers and automated re-testing triggers in our Shared Horizon action library.
Footnotes
- Nicholas Caputo, Administrative Law's Fourth Settlement: AI and the Scrutable State, arXiv:2602.09678 (2026). arxiv.org/abs/2602.09678
- Treasury Board of Canada Secretariat, Directive on Automated Decision-Making, Government of Canada Policy Suite (2026). tbs-sct.canada.ca/pol/doc-eng.aspx?id=32592
- European Union, Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (AI Act), Official Journal of the European Union (2024). eur-lex.europa.eu/eli/reg/2024/1689/oj
- U.S. Office of Management and Budget, Memorandum M-24-10: Advancing Governance, Innovation, and Risk Management for Agency Use of Artificial Intelligence, The White House (2024). whitehouse.gov/wp-content/uploads/2024/03/M-24-10
