Paying for words instead of the repair
Most AI is sold by the token, so buyers pay for text generated whether or not the job gets done. We propose paying AI agents for finished work that passes independent checks, with staged milestones so genuine partial progress still counts.
A plumber who billed by the wrench turn while the drain stayed blocked would not get paid. Buyers of AI agents accept that deal routinely: every wandering response and failed retry is billed.
Yet across the software industry, organisations buying artificial intelligence accept the exact opposite deal every single day. Vendors bill by the chunk of generated text, known in the trade as a token. Every wandering response, circular planning step, and hallucinated function call runs up the bill.1
When you measure work by words spoken rather than jobs finished, you get more words. Anthropologist Marilyn Strathern famously summarized Goodhart's Law: when a measure becomes a target, it ceases to be a good measure.2 With AI agents, paying by the token turns every hesitation into revenue for the vendor and confusion for the buyer.
When an assistant writes a software patch or files a report, the number of tokens it burned tells you nothing about quality. A model that loops four times and generates ten thousand words of boilerplate is far less valuable than one that edits three lines and passes the existing test suite.3
Researchers evaluating long tasks have documented this challenge clearly. In complex terminal benchmarks across real-world tools, agents frequently burn large token budgets looping across failed retries without finishing the job.4 Generating words is easy. Reaching resolution is hard.
Paying strictly for passed checks also has real limits. In long-horizon benchmarks, all-or-nothing binary checks scored an average of only 0.221 on reconstruction tasks, while dense progress ladders that credited incremental milestones averaged 0.653.4 Pure outcome checks discard genuine partial work. Real-world contracts need staged milestones, rewarding steady progress rather than demanding a single leap.
This is why accountability requires separating the conversation from the proof. When an agent acts on your behalf, you need an independent check that runs outside its own context window.5 Did the code compile? Did the database migration balance? Did the external message match the allowed policy?
If the answer is no, the run did not succeed, no matter how eloquent the explanation sounds. A system designed around finished work only records an outcome when an independent verification check passes. We think this simple shift changes how teams build and buy software.
Verifiers themselves are not immune to failure. In benchmark evaluations of executable verifiers across nineteen language models, balanced accuracy ranged from 10.6 to 62.1 percent, frequently rejecting valid solutions or passing broken ones.5 Independent checks must themselves be thoroughly tested and deterministic. Demanding signed receipts gives human teams a clearer view of what ran, but human judgment remains necessary.
You can explore tested execution checks and dense verification cards in our Shared Horizon library.
Related essays: When the middleman pads the meter · Why an AI agent will not audit itself · Transparency you can actually use · Who gets credit for upkeep?. Further reading: Why watermarking breaks the words · Who is actually acting?.
1. MIT Technology Review, “Stop counting tokens, start counting what ships,” November 2, 2026, technologyreview.com/2026/11/02/1147702/stop-counting-tokens-start-counting-what-ships.
2. Marilyn Strathern, “Improving ratings: audit in the British University system,” European Review 5, no. 3 (1997): 305–321; see also C. A. E. Goodhart, “Problems of Monetary Management: The U.K. Experience,” in Monetary Theory and Practice (London: Macmillan, 1984), 91–99, doi:10.1007/978-1-349-17295-5_4.
3. Carlos E. Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” ICLR 2024, doi:10.48550/arXiv.2310.06770.
4. METR, Measuring AI Ability to Complete Long Tasks (March 19, 2025), metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks; and Zongxia Li et al., “Long-Horizon-Terminal-Bench 2.0: Automating Diverse Task Construction and Measuring Sustained Agent Progress,” arXiv:2610.14146 (2026), arxiv.org/abs/2610.14146.
5. “VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents,” arXiv:2610.28679 (2026), arxiv.org/abs/2610.28679; and Stephen Casper et al., “The 2026 Singapore Consensus on Global AI Safety Research Priorities,” arXiv:2608.14611 (2026), arxiv.org/abs/2608.14611.
