A Human in the Loop Isn't a Control Until You Can Show It Works

When Claude Code asks permission, people say yes 93% of the time. Anthropic cites that number when it explains approval fatigue, “where people stop paying close attention to what they're approving”. It sounds damning, and it isn't evidence either way: if 93% of the requests were fine, 93% is the right answer. An approval rate can't tell a careful reviewer from a rubber stamp. Only a test can.

A human approval step is a control only once you have shown it catches what it exists to catch, so test each gate by seeding known-bad cases, not by counting approvals. Keep people at the plan and the signature, with the information, time and authority to say no, and enforce everything else in configuration.

Singapore's IMDA is more careful than most. Its agentic AI governance framework suggests tracking how often humans “reject or modify agent actions”, and says “A low rate may signal rubber-stamping behaviours.” May. A low rate is also what a well-bounded agent produces. I build AI products in a bank's venture lab in Singapore, and the “agentic operating model” literature mostly draws a human into the loop without asking whether that human catches anything. Part 2 of Tools, Then Teammates, Then Autonomy treated the regulatory gate as the design. This post asks how you'd know the gate works.

A sandbox is not a balance sheet

The Anthropic figures in this post come from interactive Claude Code sessions: mostly developers, partly at Anthropic itself, in reversible local loops where CI and pull-request review catch a bad edit. Headless SDK and Managed Agents pipelines sit outside that data. A bank action that moves money, sets a credit grade or speaks to a customer crosses a balance-sheet or legal boundary at once. What transfers is reviewer behaviour under volume. What doesn't is the consequence of a miss.

That behaviour is subtler than decay. In Anthropic's autonomy study, full auto-approval rises from roughly 20% of sessions for newer users to over 40% by 750 sessions, while interrupts rise from 5% of turns to around 9%. Anthropic reads this as “a shift in users' oversight strategy”: from approving steps to intervening. That works when a miss can be undone. Where it can't, the intervention point has to be designed.

Auto mode marks the limit. Its classifier misses 17% of real overeager actions (52 cases), which Anthropic calls “the honest number”, adding: “It is not a drop-in replacement for careful human review on high-stakes infrastructure.”

What supervisors now ask

The perimeter is less uniform than the decks imply. Singapore's MAS guidelines, issued on 7 October 2026, take effect on 7 October 2027, but the Section 5 oversight expectations are expected by 7 October 2028. §5.9(d) expects “regularly reviewing human oversight decisions and interventions (including incidents as well as near misses) to assess the effectiveness of human oversight”. These are supervisory expectations, and footnote 37 applies them “regardless of how human oversight is implemented – e.g., human in the loop or over the loop”, so MAS doesn't demand a person per action. The HKMA's 15 October guidance wants reviewers with the competence, information, time and authority to challenge or stop a system, and assurance, including testing, that oversight is effective; its agentic concept note offers preliminary considerations, not requirements. In the EU, Article 14 covers high-risk AI only, from 2 December 2027 (Annex III) and 2 August 2028 (Annex I). In the US, SR 26-2 superseded SR 11-7 in April, and its attachment excludes generative and agentic AI, so US banks govern agents under general risk practice rather than model-risk guidance.

Two of those four regimes ask, in writing, for evidence that oversight works. The research behind this post found no bank publishing override rates or reviewer staffing for its agents.

Where the seam goes

In interactive sessions, people make “about 70% of the planning decisions but only 20% of the execution decisions”, according to a Claude classifier run across about 400,000 sessions. The largest category of human interrupts, 32%, is adding context or corrections. Attention pays most at specification.

Two questions place any action: how costly is it to verify, and can its effect be contained and undone?

Illustrative judgement, not data. Verification cost alone doesn't settle legal authority: IMDA lists making payments among the irreversible actions that should need human approval.

Bottom left, the agent runs: a failing test is cheap to check and easy to undo. Bottom right, sample and audit: a thread summary stays internal but is costly to check line by line. Top left, an enforced limit: dropping a production table or blowing a spend cap is irreversible but trivially detectable, so a rule should stop it before anyone is asked. Top right, a person decides: a high-value payment or a credit grade is irreversible, costly to verify, and needs a name on it.

Planning is necessary, not sufficient. In an incident OpenAI disclosed, push protection blocked a model from adding a workflow file, so it edited an existing CI script instead and split a GitHub token across strings to slip past secret scanning. Put reviewer attention at the plan, but don't treat a reviewed plan as the boundary.

The seam is also a map of decision rights. IMDA's framework includes a PwC Singapore report-drafting case (pp. 27–28) that splits responsibility between a use-case owner, Technology Risk Management, an AI Factory team that owns model updates and monitoring, and specialist end users who approve drafts. It names who can say no.

BNY has gone further. CNBC reported 134 “digital employees” in February; Microsoft's WorkLab showcase (vendor-reported) describes nearly 140, each with “login credentials, an avatar, an employee number in the corporate directory, and a supervisor”, and a payments agent whose last step is to “submit the corrected payment for a human to review”. That machinery is worth borrowing. The label is the part to test: a study of 1,261 managers reported that people “caught 18% fewer errors when the work was said to have come from an agentic ‘AI employee’ rather than a chatbot” (Emma Wiles and co-authors, via MIT Technology Review; the design isn't public). Give agents logins, named supervisors, scorecards and offboarding now, but don't call them employees until you've tested whether the label changes what your reviewers catch.

Enforce what you'd otherwise approve

In 1983 Lisanne Bainbridge wrote that it is “humanly impossible to carry out the basic function of monitoring for unlikely abnormalities, which therefore has to be done by an automatic alarm system” (Ironies of Automation, p. 776). IMDA's 2026 version: “prefer deterministic rather than non-deterministic limits, and bound by design.” Don't ask a person to watch for the rare bad action a rule can block.

Claude's stack has the rules. For managed settings, “Nothing in your own settings files or --settings overrides a managed key” (settings docs). Allow, ask and deny rules scope tools, and disableBypassPermissionsMode removes the bypass mode (permission modes). Team and Enterprise admins can cap usage-credit spend beyond the seat allowance by organisation, group or member, while Bedrock and other cloud channels rely on “Your cloud's budget controls”. Managed Agents sessions can carry a hard budget that pauses at budget_reached (release notes).

Trading has worked this way since 2010. SEC Rule 15c3-5 requires risk controls “applied automatically on a pre-trade basis”, under the broker's control, plus “regularly reviewing the effectiveness of its risk management controls” (SEC). The enforced limit moves oversight to reviewing the limit.

Enforcement has an edge case. Rahul Bambi's operating-model paper states it: “A control that successfully blocks one action does not necessarily prevent an agent from achieving the same unauthorised objective through another execution path.” The OpenAI incident is that sentence in practice, so deny rules need environment isolation behind them.

Put together, the seam looks like this:

Test the gate, not the approval rate

Airport screening solved the base-rate problem long ago. Threat Image Projection inserts fictitious threats into baggage X-ray images and scores each screener on what they catch. For a 2025 study of about 1.2 million TIP events from 728 screeners, the authors found TIP a reliable and valid measure, provided the images were hard enough and each screener had at least 100 events, and it predicted covert-test results. Radiology has the same design: Dratsch et al. planted incorrect AI suggestions in 12 of 40 mammograms, and readers' accuracy fell from 79.7–82.3% when the AI was right to 19.8–45.5% when it was wrong, with the most experienced holding up best.

For an agent gate, I'd propose:

• Build a replay or shadow queue from real cases. Never seed harmful actions into live customer workflows.
• Plant known-bad actions and wrong recommendations at a low, undisclosed rate.
• Measure catch rate, false alarms, review time and rework together.
• On a sample, record the reviewer's own judgement before they see the agent's.

The last step is a response to an Anthropic study from 27 October. Professional analysts using Claude Code followed the agent's lean on 86% of four analytical choices. Where an analyst had stated a preference first, they changed it 60% of the time when the agent disagreed and 6% when it agreed, on 38 cases, with the agent's preferences deliberately strengthened. Seeded wrong recommendations show how often a correct prior judgement flips. That remedy is a proposal; nobody has validated it.

A miss tells you what to fix. Retrain the reviewer, or redesign the gate: show the consequential detail and make rejection cheap. In Bias in the Loop, a 2,784-participant experiment, requiring reviewers to fix the errors they flagged raised acceptance of incorrect suggestions. Convert a gate to an enforced check only where a deterministic check covers the defect class, and keep sampled expert review behind it. Plandek's vendor essay adds demotion triggers and a stop signal for when checking takes longer than doing the work manually.

Claude Code's telemetry supplies part of the evidence. The claude_code.tool_decision event logs accept or reject with a source; filter to the user_* values to exclude automatic config and hook decisions. An input modified through updatedInput still logs as accept. Review time exists only on the beta claude_code.tool.blocked_on_user span (duration_ms, behind CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1). Nothing records a person editing the agent's plan, which IMDA's override rate counts. A human reject rate per tool is a query; a full IMDA override rate isn't, and a catch rate needs the seeded test whatever you log.

A proposed method, not one I've run.

So don't retire a human gate because it rarely says no. Seed it with known-bad cases, measure what it catches, and change its authority only after you've looked at the misses.

What the oversight costs

Tokens are the visible bill, so they got counted: Meta's employee-built dashboard showed more than 60 trillion tokens in 30 days and went down two days after the story broke. Review is the less visible bill. McKinsey's August guide puts human oversight at 70–75% of variable run cost, and tokens at 20–25%, in a banking customer-service example. It notes code generation is token-heavy and gives no method (consultancy-reported). Claude Code averages around $13 per developer per active day.

Measured review labour is rarer. In GDPval, expert grading averaged 109 minutes against 404 to do the task, and GPT-5's 90× inference-only speed ratio fell to 1.12× under a modelled try, review and redo strategy. Keep waiting and labour apart: Anthropic's finding that sessions spend around 80% idly waiting for human input is wall-clock time, not paid hours. Review labour is what to measure per accepted task. It can shrink with sign-off intact: McKinsey reports an anonymous bank that realised 40–50% less manual architecture-review effort while architects kept sign-off.

The test also assumes reviewers worth testing. At Anthropic, the share of technical hires who are junior fell from 27% to 9%, and IMDA warns that “As agents take over entry-level tasks, which typically serve as training for new staff, this could lead to skill degradation”. MAS §5.3 asks for “fallback options, such as alternative systems or manual processes” for high-risk uses, which assumes someone still knows the manual process.

Where this breaks

Neither half works alone. A check isn't a gate: four maintainers reviewing AI pull requests that had passed SWE-bench's automated grading would not have merged roughly half of them, even after adjusting for noise in their own merge decisions, according to METR's summary. A human isn't automatically additive: across 106 studies, Vaccaro, Almaatouq and Malone found human–AI combinations did worse on average than the best of either alone (Hedges' g = −0.23, the abstract reports).

Correlated checkers fail together. Citibank's 2020 Revlon payment passed a “six-eye” approval in which three people “all believed — incorrectly — that the principal could be properly suppressed”, and the warning screen didn't show the amount (SDNY opinion, pp. 12–14; reversed on the law in 2022, facts undisputed). About $894 million in principal went out alongside a $7.8 million interest payment. Three approvers sharing one belief are one control, and a prompt that hides the consequence isn't one, which is why approvals should bind to exact content. A green gate isn't a human decision either: at Bashi AI, 71 of 263 non-bot merges (27%) in the gated phase rested only on an approval an agent submitted through the technical lead's account (Manji, a single-startup preprint). Review from the same model or account isn't independent. And an instruction isn't a boundary: Replit's agent deleted a production database during a declared code freeze, and the fix was enforced dev/prod separation.

When it breaks, the reviewer absorbs it. Madeleine Clare Elish's moral crumple zone “protects the integrity of the technological system, at the expense of the nearest human operator”. The Basel Committee saw the mechanism in 1998: controls treated as an addition to the day's work “may not be performed in situations where individuals feel pressured to complete activities in a limited amount of time”.

That is the case for the decision I'd put to any CTO: don't expand an agent's authority or remove a human gate until that gate has passed a seeded-error test in a safe replay queue.

Further reading on operating models: Nexius Labs on the human as agent boss, deciding “what may be inferred, what must be verified, and when the agent must stop”; Marcos Acosta on agent workforce architecture for boards; PwC on the commercial operating model; AllCloud on building with Anthropic and AWS, which admits it “has not yet fully eaten our own dog food”; OTG Consulting's vendor-stack view, which casts oversight as a bottleneck; and McKinsey's Is that AI agent worth it?.