The Agent Timed Out. Did the Action Happen?
$ grep -n "^##" 2026-10-agent-timeout-safe-retry.md
A timeout alone cannot establish that a remote action failed.
Consider a fictional agent creating a ticket for incident INC-42. The provider creates the ticket, but the response never reaches the caller. The agent sees a timeout. Asking it to “try again” could create another ticket unless the provider recognises the retry as the same operation.
Acknowledgement and completion are different observations. Losing an acknowledgement creates another problem: choosing the next action without knowing what the first one did. A safe decision needs either sufficiently authoritative outcome evidence or a provider contract that makes replay safe.
Keep the intent when the attempt changes
For this example, the intent is to create one ticket for INC-42. An attempt is one submission of that request. A timeout changes what the caller knows; it does not create a new reason to open another ticket.
AWS’s explanation of idempotent APIs describes caller-provided request identifiers for this purpose. It also explains why matching parameters are insufficient: two identical requests can represent two intentionally separate resources.
Before submitting, I would persist a small operation record:
intent: create one ticket for INC-42
target: support account / production
operation: create-ticket
provider_key: <opaque key retained across attempts>
request: <durable reference to the original parameters>
outcome: not_submitted
This is a proposed record, not a provider API. Before issuing the request, durably mark submission as started with its outcome unresolved. After a crash, that state requires reconciliation or contract-covered replay; it is not evidence that nothing was sent. Attempts and observations attach to that same intent. A deliberate second ticket gets a new intent and key, even if its content is identical.
The record preserves enough information to make a decision after a restart. It cannot make a remote write atomic with the local database. There is still a possible crash after the provider acts but before the caller records success.
Choose the next action from the evidence
Use the following decision table for the hypothetical ticket workflow. “Confirmed” means evidence for this operation, in the correct account and environment, with the provider’s documented semantics.
| Observation | Next action |
|---|---|
| Provider confirms this operation completed | Store the ticket ID and completion evidence; continue |
| Provider reports it is still running | Observe within a bounded waiting policy |
| Provider confirms rejection before execution | Correct the request within existing authority and follow its retry rules |
| Outcome is unknown, but replay remains covered by a valid idempotency contract | Replay the original request with the same key, or continue reconciliation |
| Outcome is unknown and no applicable replay guarantee remains | Stop writes and escalate reconciliation |
The fourth row matters. You do not need a separate status endpoint before every safe retry. Replaying an idempotent request can itself recover the response. Requiring a lookup first would add a dependency that the provider’s contract may already make unnecessary.
Conversely, an empty search result is not enough if that search can lag behind writes. The original request might still finish, or the resource might not yet appear. A ticket with the same title is also weak evidence: another caller could have created it. Reconciliation needs a reliable association with the original intent.
Where the API supplies no such association or replay guarantee, retain the uncertainty. Renaming unknown to failed does not make another write safe.
Read the contract behind the key
Stripe’s idempotency documentation gives a useful concrete example. Stripe saves the first executed request’s status code and body for a key, including a 500 response. Subsequent requests with that key return the saved result.
That means a retry can be safe from duplicate execution without producing a successful outcome. A repeated 500 is not an instruction to generate a fresh key and try the action again.
Stripe also checks the incoming parameters against the original request. Validation failures and conflicts with concurrently executing requests do not save an idempotent result, because endpoint execution has not begun. Those exceptions belong to Stripe’s documented behavior; they are not permission to assume that every error response means nothing happened.
Retention is another boundary. Stripe says keys can be removed after they are at least 24 hours old, and reuse after pruning generates a new request. That is neither an exact expiry time nor a guarantee shared by other providers. AWS’s account of idempotent API design explicitly describes retention requirements as service-specific.
For each endpoint, establish the applicable caller or account scope, operation, parameter rules and retention guarantee. Keep retries inside that contract. A key stored indefinitely in your workflow database does not extend the provider’s memory of it.
Bound retries without discarding uncertainty
Suppose the ticket response remains unavailable after several attempts. Give the workflow a retry budget and an escalation condition. Google’s SRE guidance describes retry budgets and warns about retries at multiple layers; an agent loop should account for retries already happening inside its SDK.
Budget exhaustion changes who must resolve the operation. It does not establish that the ticket was never created.
For the example above, a useful handoff retains the original intent, target account, request parameters, provider key, attempt times and every returned operation or resource ID. It identifies the unresolved question: whether that specific creation completed. If contract-covered replay is still available, say so. If it is no longer available, keep further creation blocked while the outcome is investigated.
Idempotency cannot establish that the ticket was authorised or addressed to the right account. Those checks belong before submission. After a lost response, preserve that original authorised intent and resolve its outcome before turning uncertainty into another action.
$ subscribe --newsletter
Practical AI engineering, in your inbox
Field notes for technical leaders building agents, evaluation systems, governance, and production infrastructure.
Related
Your Fitness Agent Needs Reliable Data Before Better Advice
A technical follow-up to the Reverse Tamagotchi: physiological days, missing scores and stale rows can change what a fitness assistant thinks it knows.
When Should an Agent Ask Permission?
A useful agent acts within delegated authority, prepares consequential actions for review, and asks again when the action changes.
The 5-Step Loop: Why Your Agent Fails at Step 4
ReAct gave us a three-step loop. Production hardened it into five. The two new steps — Plan and Verify — are where everything that goes wrong, goes wrong. And the field has now named the worst offender.