When software makes a decision about you, what should you be able to see?
When software shapes a decision about your housing, job, credit or benefits, you should be able to see that software was used, which facts it relied on, what would have changed the outcome, and how to reach a person who can fix a mistake.
A rental application is turned down in thirty seconds. A rehabilitation claim is denied overnight. When you call to ask why, the clerk's screen doesn't say either. Most people don't object to software handling routine paperwork. They object to life-shaping decisions that nobody can explain or correct.
We have been here before
Half a century ago, credit bureaus began collecting household dossiers that decided who could get a mortgage, an apartment or a job. Choices that once happened across a local bank desk moved into distant filing cabinets.
Families regularly lost out on homes because of a simple clerical error inside a folder they were never allowed to inspect. A neighbour's unpaid utility bill got filed under the wrong middle initial, or a debt paid off years earlier stayed marked as delinquent.
Now contrast that locked cabinet with a municipal building inspector's checklist. When an inspector checks the wiring or stairs in a house and marks an item as failed, they do not hand you a mysterious score out of a thousand.
Instead, their checklist points to the exact rule that was missed, such as a handrail two inches too low. It tells you exactly what to fix before they return. You can verify the measurement yourself with a tape measure.
Laws like the United States Fair Credit Reporting Act of 1970 pushed credit files closer to that inspector's checklist.1 They did not ban credit scores; instead, they established a basic right to see the file, know the main reasons for an adverse decision, and correct untrue records.
Today's screening software is faster and more complex than a room full of paper folders. Yet the human problem is identical. An opaque cabinet is still an opaque cabinet, whether it is built out of steel drawers or neural network weights.
Four things you should be able to see
What does that building-inspector bargain look like when software makes or shapes a consequential decision about you today? We think every person should be able to see four straightforward things.
1. Plain notice that software was involved. You should never have to guess whether a human reviewer read your file or an automated program filtered it out before anyone looked.
That disclosure does not belong buried on page forty of a general terms-of-service document. It belongs right on the letter, email or screen where the decision is delivered, alongside the name of the system used.
2. The specific facts the system used about you. Many "algorithmic errors" in everyday life are mundane. A stale eviction record belongs to someone with a similar surname, a date was misread on a scan, or a tax form went missing and nobody asked you to check.
No formula on earth can produce a fair answer if the facts fed into it are wrong. Before arguing over fifty variables, you should receive a clean copy of the records used about you. You also need a direct way to correct false items.
3. What would have changed the outcome, in plain language. Organisations sometimes claim that modern machine learning is too complex to explain to an ordinary citizen, or that explaining a score would expose proprietary trade secrets.
You do not need the company's source code or a spreadsheet of ten million neural weights. Like the homeowner reading the building inspector's checklist, you need an honest, checkable reason: which specific factor mattered most, and what would need to change for the answer to become "yes."2
Imagine an apartment notice with an exact reason: "Your application was declined because our screen showed ten months at your employer, while policy requires twelve." You can check that statement in ten seconds.
If the tenant-screening broker scraped an old record and you have actually worked there for fourteen months, you know which pay stub to upload. If the ten-month figure is accurate, you know immediately whether adding a co-signer solves the problem.
4. A doorway to a real person with both the time and the authority to overturn a mistake. Many institutions boast that they keep a person in the loop. Too often, that means a call-centre worker given ninety seconds per call. Their screen is greyed out, with no button to override the software.
An appeal to a representative who may not overrule the computer gives you no real remedy. Real contestability means reaching a reviewer who has both the time to read corrected evidence and the institutional authority to reverse the decision.
The four questions
When software shapes a decision about you, ask: Did software score or filter this? What specific records about me did it use? What would have changed the answer? And who has the authority to overturn a mistake?
Walkthrough: two mornings with the same mistake
To see why these four pieces must work together, picture a nurse named Elena applying for a flat near her hospital after moving across the province. A data broker mixes up her file with another person's old utility arrears from a city she has never visited.
In the locked-courthouse version of that morning, the property manager's portal rejects Elena in twenty seconds with a generic message saying her composite tenant score fell below the required mark. The leasing agent shrugs and says the software decision is final. Elena loses the apartment and still has no idea a stranger's unpaid bill is attached to her name, waiting to sink her next three applications.
In the contestable version of that morning, the rejection screen attaches a signed decision receipt. Elena sees immediately that an automated screening tool ran, spots the unfamiliar utility account listed under "Inputs Used," and clicks the appeal link to attach her actual utility history. A human leasing supervisor reviews the mismatch that afternoon, clears the false record, and approves the lease.
Rules you can already rely on
None of these four expectations is wishful thinking. Across several jurisdictions, citizens can already invoke existing rules, or press their representatives to close the loopholes that remain.
In the United States, the Consumer Financial Protection Bureau has stated clearly that lenders cannot hide behind "complex algorithms" or black-box models to avoid giving specific, accurate reasons for denying credit.3 Even when a machine-learning system makes the call, the lender must name the principal factors that drove the outcome.
In Canada, the federal Directive on Automated Decision-Making governs administrative systems. Departments must publish an impact assessment, provide plain notice, and explain how a decision was reached.4
In the European Union, data protection law restricts significant decisions made solely by automated processing, establishing rights to obtain human intervention, express a point of view, and contest the decision. Still, organisations often try to sidestep those protections by placing a nominal rubber-stamper at the end of the queue.5
Where the biggest gap remains is in everyday scope and enforcement. Rules that cover a bank mortgage or federal pension are often weaker when software screens job applicants, scores tenants, or flags insurance claims.
An honest look at the limits
We also need to be candid about where explanations fail on their own.
First, handing someone an explanation only helps if the record cannot be rewritten afterwards. If a system turns you down and invents a tidy excuse when you complain later, the explanation is theatre.
Everything the software used at the moment of the decision (the inputs, the model version, and the explanation rule) must be sealed into a receipt right then. That way, an auditor can replay the check and confirm the reason was genuine.
Second, a right to ask questions fails if people are punished for asking. If a renter fears that challenging a screening score will land them on an informal blacklist, most renters will stay silent. Contestability requires clear rules against retaliation.
A step you can take this week
Next time an employer, insurer or agency turns you down using automated software, ask in writing for two things. Request a copy of the exact records used, and ask for a review by a person authorized to correct errors.
If you belong to a union, tenant association or neighbourhood co-op, ask those questions together. One applicant is easy to brush off with a form letter. A tenant union or workplace group asking an institution to audit its screening tools is much harder to ignore.
You can find practical templates on our For everyone page, and briefing checklists on our For policy makers page. Software can help institutions move faster, but it should never become a locked door that lets them stop answering to the public.
1. United States Fair Credit Reporting Act (1970), 15 U.S.C. § 1681 et seq., Federal Trade Commission Legal Library: ftc.gov/legal-library/browse/statutes/fair-credit-reporting-act.
2. Sandra Wachter, Brent Mittelstadt and Chris Russell, Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR, Harvard Journal of Law & Technology, Vol. 31, No. 2 (2018).
3. Consumer Financial Protection Bureau, Consumer Financial Protection Circular 2022-03: Adverse Action Notification Requirements in Connection with Credit Decisions Based on Complex Algorithms (May 2022).
4. Treasury Board of Canada Secretariat, Directive on Automated Decision-Making (Policy ID 32592).
5. European Union, Regulation (EU) 2016/679 (General Data Protection Regulation), Article 22 (Automated individual decision-making, including profiling).
