Human in the Loop is Fiction

Addressing the Fiduciary Responsibility

AI Agents are everywhere. Already we've seen model-powered agents analyzing, refactoring, testing and monitoring key digital assets of companies, from billing and financial reporting to customer tranasactions and human resources. They operate at machine speed, sending hundreds, then thousands, of notifications a minute to the people who operate at human speed, and are nominally responsible for them. The board still owes shareholders what Delaware courts call a good-faith effort to implement an oversight system and then monitor it. Since Marchand v. Barnhill, that duty is sharpest for risks that are "mission critical."¹ Billing, financial reporting and order approval are exactly that. So the question isn't whether a "human is in the loop". It's whether that human understands the loop.

The Irony Returns

Margaret Mitchell, Chief Ethics Scientist at Hugging Face, and her co-authors argue in a new position paper that today's agent systems don't support human oversight — they degrade it.² Agent design hides the moments that need judgment, and extended use of AI erodes the very skills an overseer needs. It is an old lesson in new clothes. In 1983, Lisanne Bainbridge's "Ironies of Automation" warned that automating a job leaves people the hardest part — catching the rare failure — while removing the practice that keeps them able to catch it.³ Medicine has already run the experiment: clinicians override 49 to 96 percent of drug-safety alerts.⁴ And a 2025 survey of 319 knowledge workers found that the more people trust generative AI, the less critical thinking they apply.⁵ A thousand alerts a minute is not oversight. It is noise with a signature line.

This, more than model quality, explains the drag on production-first adoption of AI in critical systems. The agents are fast and often right. What's missing is credible, demonstrable supervision by the functional experts accountable for the outcome.

What We Learned Building With Agents

At Strategic Machines, crews of agents design, build and release software for our clients. Mitchell's paper names three remedies. We arrived at the same three by necessity.

Strategic friction. Before work starts, the owner states the outcome, the acceptance criteria and what must never change — in their own words. Approval is given against the exact version shown; if the design changes afterwards, the approval is voided, not carried forward. Decisions that move money or touch customer data need a deliberate second confirmation. A little effort, placed exactly where it forces a person to think.

Decision design. Expert attention is the scarcest resource in the enterprise, so we ration it. Every request is rated by what is at stake. A typo ships on its own once every test passes. A new feature needs a one-page brief. Payments, personal data, sign-in and security stop for the accountable owner at the design, the plan, the preview and the release — and only there. When we added Stripe checkout to a test store this week, the owner made five decisions while the test suite grew from 243 to 276 checks, every one run at every step.

Behavioral monitoring. We treat oversight as something to measure, not assume. Every decision is recorded with who made it, on which version, how and when, and every test run is bound to the exact code it judged. The agent that writes the code never grades it. That record answers the questions an auditor will ask — and shows when approvals start arriving faster than anyone could have read what they approved.

Tether, Then Trust

"Human in the loop" was a reassuring phrase for a slower era. With hundreds of agents, it is fiction unless the loop is engineered: fewer, better decisions, each bound to evidence, each provably made by someone qualified to make it. Regulators already see it — the EU AI Act requires that overseers remain aware of their own tendency toward automation bias.⁶ This is not a feature request. It is a redesign of how authority, evidence and accountability flow through the enterprise.

In Part 2 of this series, Managing Complexity, Not the Code, we turn to the people who build digital products — and why their role is changing faster than their titles.

At Strategic Machines, we deliver the Accelerator: an expert crew of AI agents, trained for your industry, that improves your digital products under your direction. Every change arrives as a defined work package — the request, the design, the plan, the tests, your approvals and the release — kept together as a lasting record you own.

See it in practice: a shop owner adds Stripe checkout to her store, from request to release, with every decision on the record.

Let's talk.

SOURCES AND REFERENCES

¹ Marchand v. Barnhill, 212 A.3d 805 (Del. 2019), applying In re Caremark International Inc. Derivative Litigation (Del. Ch. 1996). See Jones Day, "Delaware Supreme Court Reinforces Directors' Oversight Obligations on Mission-Critical Subjects" (2019).

² Margaret Mitchell, Avijit Ghosh and Samir Passi, "AI Agents Push Humans Out of the Loop," arXiv:2608.23642 (2026). Mitchell is a researcher and Chief Ethics Scientist at Hugging Face.

³ Lisanne Bainbridge, "Ironies of Automation," Automatica 19(6), 1983.

⁴ Heleen van der Sijs et al., "Overriding of Drug Safety Alerts in Computerized Physician Order Entry," Journal of the American Medical Informatics Association 13(2), 2006.

⁵ Hao-Ping (Hank) Lee et al., "The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers," CHI '25, Microsoft Research and Carnegie Mellon University, 2025.

⁶ European Union, Artificial Intelligence Act, Article 14: Human Oversight, paragraph 4(b).