How to Measure the Human Cost of Agent Work

Before budgeting the time an agent saves, measure the human work required to get an acceptable result.

In 26 Minutes vs 33 Seconds, I argued that assumed oversight time is a poor basis for capacity planning. This is the collection protocol I would use to replace that assumption. It is a proposed method, not a report of an experiment I have completed.

The immediate question is modest: how much human effort and agent spend does this workflow consume per completed task, at an agreed quality bar? Proving it saves time against another method requires an additional comparison.

Define the task before starting the timer

Choose one task class: a bounded bug fix, a source-backed research note, a routine data reconciliation. Write down eligibility rules and register every selected task before work starts. Do not collect only the runs somebody considers worth sharing.

For each task, specify the required result, who can accept it, and what evidence they need. Fix the collection cutoff and a later defect-review date. Acceptance today and a discovered defect tomorrow must be able to coexist in the record.

Use the business task as the unit, keeping every agent attempt beneath it. A rejected first patch followed by two revisions is one task with three attempts. Starting a new chat does not reset its accumulated cost.

Give each candidate attempt its own disposition: accepted, rejected or not reviewed, with a reason and evidence. Report accepted agent candidates divided by evaluated agent candidates separately from task completion, and disclose unreviewed candidates. That yield describes the submitted artifacts, including repeated attempts on the same task.

In this protocol, a task can also finish after a human takes over. Count that completed task, but label its completion mode explicitly. You are measuring the whole workflow, including rescue work, rather than claiming every accepted result was an accepted agent artifact. Report manual takeovers separately.

Keep three clocks separate

Record active human time in categories that explain where capacity goes:

Pause the human timer while the agent runs unattended or while you do unrelated work. Task turnaround ends at its terminal outcome: accepted, rejected or abandoned. Record that time separately from each candidate’s disposition. A rejected patch does not stop the task clock if another attempt or manual takeover follows. For pending tasks, report age at the cutoff. Later defect repairs do not move the original endpoint. Waiting in a review queue affects turnaround even when nobody is actively working.

Count all reviewers’ person-minutes. Two people reviewing together consume two people’s time. Conversely, one person cannot log the same five minutes against three agents and call it fifteen minutes of active work. Split or label shared intervals using a consistent rule.

Record agent execution duration if it helps diagnose latency, but do not add parallel runs together and call the result elapsed turnaround. Human effort, machine runtime and delivery time answer different questions.

A missed timer entry is missing data. Mark it as such; an unrecorded review is not a zero-minute review. If you reconstruct an interval afterwards, label it estimated.

Keep shared human setup and maintenance in a separate cohort overhead record. Report that effort alongside task execution. If allocating it across tasks, declare the rule and count it once; distinguish task-only from overhead-inclusive costs before deciding whether delegation is worthwhile.

Attach spend to every attempt

Include generation, retries, tool charges and automated verification attributable to the task. An evaluator agent also costs money. Keep failed attempts attached to the same task ID so changing models or sessions does not make them disappear.

Prefer actual provider charges where available. If you estimate from usage, retain the pricing date and relevant billing assumptions, including cache treatment. Keep currency explicit and reconcile the estimate when billing data arrives.

A flat subscription does not produce a precise marginal charge per task. You can report it separately or allocate it by a declared rule, but label the allocation. Apply the same discipline to shared infrastructure. If you convert human minutes into money, state the hourly-rate assumption separately from observed effort.

Let failures affect the denominator

For a closed cohort with complete effort and charge records, calculate:

human minutes per accepted task = all cohort human minutes / accepted tasks
agent spend per accepted task   = all cohort agent spend / accepted tasks
acceptance rate                = accepted tasks / started tasks

“All” includes review and rework on rejected or abandoned tasks. “Accepted” means the agreed result was achieved; split the count between agent-assisted completion and manual takeover. Otherwise a successful rescue can look like successful delegation.

Here is synthetic arithmetic, not collected data: three started tasks consume 45 human minutes and $6 in agent/tool charges. Two meet the acceptance criteria; one is abandoned. The workflow consumed 22.5 human minutes and $3 per accepted task, with two of three tasks accepted. Dividing only the successful tasks’ costs by two would hide the abandoned work.

With zero accepted tasks, the per-accepted-task ratios are undefined. Report the spend, effort and outcomes directly.

If tasks remain pending at the cutoff, show their accrued effort and spend, their count and how long they have been open. Any current unit-cost ratio is provisional. Do not present the easy tasks that finished first as the final cohort result.

Missing measurements also prevent a complete unit-cost claim. Report observed subtotals, estimated amounts and which tasks have missing intervals or charges. A ratio using only observed effort is a lower bound, not the complete cost. Keep those tasks in the outcome counts.

Freeze totals at the collection cutoff. Keep later work linked to the task, identifying defect repairs through the follow-up window. Report additional effort and charges separately; a cumulative follow-up total adds that increment once. Preserve the original acceptance decision alongside any later defect.

A baseline answers a different question

This record can expose expensive review or frequent rescue without proving an agent caused either. To claim savings, compare comparable tasks under the same acceptance bar, with a reviewer qualified to apply it.

Where practical, assign eligible tasks to the agent-assisted and comparison workflows before execution, using randomisation within task classes. METR’s early-2025 developer study assigned real issues to allow or disallow AI; its methodological relevance here is that assignment preceded the outcome. Its findings belong to that sampled setting, not every later tool or team.

Avoid doing a task with an agent and then timing yourself doing it manually. You already know the solution. Keep task mix, developer familiarity and quality requirements visible; show individual records and distributions when a small sample makes an average misleading.

The SPACE framework treats developer productivity as multidimensional. Time and cost should sit beside acceptance and defects, not replace them. A shorter review can mean an easier artifact or a weaker review; the timer cannot distinguish those by itself.

A record you can use

This blank template uses one task-level ledger for human time. An optional attempt ID links review to a candidate; briefing and manual rescue need no invented candidate. Each person-minute appears once. Derive category totals and cutoff snapshots from the segments, keeping measured and estimated time separate. Nulls mean unmeasured or unassigned, not zero.

task_id: null
task_class: null
acceptance_criteria: null
reviewer: null
workflow_version: null
started_at: null
disposed_at: null # terminal task outcome; null while pending
collection_cutoff: null
defect_followup_until: null
attempts:
  - attempt_id: null
    candidate_revision: null
    model_tool_versions: null
    disposition: null # accepted, rejected, or not_reviewed
    reason_and_evidence: null
human_segments:
  - segment_id: null
    person: null
    attempt_id: null # optional
    category: null # briefing, supervision, review, rework, verification
    phase: null # initial_work or defect_followup
    started_at: null
    ended_at: null
    basis: null # measured or estimated
charges: [] # timestamp, optional attempt_id, phase, amount, currency, basis
missing_records: [] # affected work/charge and what remains unknown
totals_at_cutoff: null # derived category totals by basis, plus spend
additional_after_cutoff: null # same shape; excludes cutoff totals
outcome: pending # accepted, rejected, abandoned, pending
completion_mode: null # agent_assisted or manual_takeover
acceptance_evidence: null
followup_defects: [] # discovery time, evidence, linked repair segment IDs

Before collecting, decide what would make this task class worth delegating: an acceptable human-time budget, quality bar and tolerance for manual takeover. Then use the records to choose whether to continue, change the workflow or stop delegating that class. Knowing where your reviewers’ time went is the first result you need.