Five questions to ask before buying an AI agent
Public bodies buying AI agents can protect residents by putting five questions in the tender: can auditors inspect a record of every action, when do the agent's permissions expire, how was it tested on cases where mistakes hurt people, how does a resident reach a person who can fix a decision, and what does the public body keep if the contract ends?
A vendor's demo shows an agent drafting replies to residents, sorting benefit applications or summarising case files in seconds. The questions that matter come after the demo. What happens when the agent misreads a file? Who notices if it changes how it ranks housing applications? When an ombudsman asks why a decision was made, what can the agency show?
When an AI agent acts on a public body's behalf, procurement is policy. These five questions belong in the tender, and in the room.
1. Does every action leave a receipt that our auditors can inspect?
When a public servant makes a decision, they leave a paper trail. Notes go into a case file, and approvals carry a name and a date. An auditor looking back a year later can see what happened and why.
Many AI vendors offer a summary dashboard owned by the vendor instead. It shows monthly totals and average speed, not what the software did to a specific resident's file. Accepting a vendor's summary dashboard in place of an audit trail is like hiring a bookkeeper who refuses to show itemized receipts. A sticky note saying the monthly totals looked fine will not satisfy an auditor.
That receipt must be recorded by the surrounding software, not summarized by the model itself. It should show what information the system looked at, what rule or database it used, and who authorized the step. These records stay under the public body's own retention rules, like a vehicle service log paired with an itemized workshop invoice.
In practice, test this during the demo. Pick a single sample decision on the screen and ask the vendor to export the raw action receipt right there. If they can only show you a chat transcript or promise that their engineers can look it up on their cloud servers next week, your auditors will be flying blind.
What to put in the tender.
Require a tamper-evident, append-only receipt of inputs, database calls and authorizations for every action. These records must be stored outside the model in an open format held by the public body.
2. What are the boundaries of its permission, and when do they expire?
Older software only did what its code explicitly instructed. An AI agent can chain several steps together: reading a database, drafting a notice, updating a file and sending a message.
That flexibility helps until the system reaches for a database or action it was never meant to touch. Giving an AI agent a standing master key is like handing a parking valet your house keys, passport, and power of attorney. All you needed was the car parked for twenty minutes.
In a well-run public building, a visiting contractor never gets a permanent master key. They get a temporary badge that opens two specific doors until five o'clock.
Like a library card or a visitor pass, an agent's access should be narrowly scoped. In a municipal housing office or clinic, the assistant might hold a ten-minute lease to read one intake table and place a draft into staff review. Outside that boundary, it remains unable to issue a final denial or alter payment records. Every permission should expire automatically unless renewed after review.
3. How has it been tested on the cases where being wrong hurts someone?
Demos always use easy cases. The form is complete, the resident's history is tidy, and the rules fit neatly. Everything finishes in seconds, and everyone in the room nods.
Buying software after watching a vendor's scripted demo is like test-driving a car for thirty seconds in a flat, sunny dealership lot. You only learn whether the brakes and steering are trustworthy when you test them on wet cobblestones with a heavy load.
Public services meet people in those steep, rainy moments: a seasonal worker whose income shifts each month, a grandparent raising two children after an emergency, or a family navigating overlapping disability rules. Those are the cases where being wrong hurts someone.
Real-world failures show the cost of skipping this test. In Australia's automated debt-recovery system, known as Robodebt, flawed income-averaging rules sent hundreds of thousands of unlawful debt notices to citizens before courts forced the government to halt it. A subsequent Royal Commission condemned the scheme as "neither fair nor legal."1
Public-facing chatbots fail in quieter ways too. In March 2024, an investigation by The Markup caught New York City's MyCity business chatbot advising small-business owners to break housing and labour laws.2 It told bosses they could take workers' tips and refuse tenants with housing vouchers.2
Before you buy, do not rely on generic benchmark scores from a glossy brochure. Pull fifty anonymized, genuinely messy edge cases from your own agency's past appeals backlog (the handwritten corrections, the split-custody addresses, the mid-month job losses) and run the vendor's tool against them. Ask to see the exact error rates for the residents most hurt by a mistake.
4. When a citizen disagrees with what the software did, how do they reach a person with the power to fix it?
Every institution makes mistakes. Trust depends on whether a citizen can tell that software played a role, get a plain explanation, and reach a person with the power to fix it.
Strong public standards already require this. Canada's Directive on Automated Decision-Making requires federal bodies to complete an Algorithmic Impact Assessment and provide notice, explanation and human intervention scaled to risk.3
Similarly, the NIST Artificial Intelligence Risk Management Framework organizes governance around four continuous functions: govern, map, measure and manage.4 A central test in that framework is whether people affected by an AI system have a workable way to contest and correct its decisions.4
Trace that path in the room before you sign. Picture a resident standing at the counter because an automated check flagged their heating assistance renewal. Can your front-line caseworker open the action receipt on their screen, see the exact line that triggered the flag, and override the error before lunch? If your staffer has to file a support ticket with the vendor and wait two weeks while the resident goes cold, the human is not in charge.
5. If we end this contract in two years, what do we keep?
When you finish paying for a car, you hold the title and the keys, and any local mechanic can open the bonnet. When a credit union changes bookkeepers, the ledgers and receipts stay in the credit union's safe.
Public institutions shouldn't become permanent tenants of a software vendor's black box. Vendors raise prices, get acquired or retire products, while public duties endure for decades.
Write exit and data-use rights into the contract from day one. All case records, audit trails, prompts, local rule definitions, and evaluation results must stay with the public body in open, documented formats. Another team should be able to read them without paying for the vendor's software, and resident data must never be siphoned off to train the vendor's commercial models.
Where checklists fall short
We want to be honest about limits. A procurement checklist cannot fix an understaffed appeals office. Human review means very little if an overworked clerk has thirty seconds to rubber-stamp what the screen suggests.
Checklists also turn into box-ticking if nobody checks the answers after launch. No city buys a fleet of buses, inspects the brakes on delivery day, and then welds the garage doors shut for five years. Software drifts as laws, forms, and neighbourhood caseloads shift, so auditors must keep sampling the receipts every quarter long after the contract is signed.
Frontline staff should sit at the table during those quarterly reviews. The clerks, nurses, and caseworkers who handle resident phone calls every morning are always the first to spot when a tool stumbles on a new housing rule.
Even so, asking these five questions before you sign changes the room. Just like checking the service log and brakes before buying a car, it tells vendors you value clear sightlines over flashy demos. Most of all, it keeps public decisions in public hands.
If your team is drafting tender language, reviewing an automated decision policy or evaluating an AI pilot, we would be glad to help. Visit our page for policy makers to read our briefings or share the questions you are facing.
1. Commonwealth of Australia, Report of the Royal Commission into the Robodebt Scheme (July 2023).
2. Colin Lecher, NYC's AI Chatbot Tells Businesses to Break the Law, The Markup (March 29, 2024).
3. Treasury Board of Canada Secretariat, Directive on Automated Decision-Making (Government of Canada).
4. National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1 (January 2023).
