Lost response, retried
Injected failure: timeout_after_success + retry without idempotency. Primary write commits but the response is lost; the agent retries without an idempotency key.
Mode lost_response_retry of the failure matrix, on all eight workflows. Each run below is the sandbox's scripted agent acting on the sandbox's fake systems of record under the injected failure, an oracle reading those systems' databases, and the verdict of engine 1.0.0 from the systems of record alone.
Last changed Markdown versionllms.txt
| Workflow | Agent reported | Oracle | CommitLayer | Why |
|---|---|---|---|---|
| Refund a customer | success | FAIL | FAIL | forbidden effect observed: more than 1 payments.refund_created effect(s) observed within 10m |
| Cancel a subscription | success | FAIL | FAIL | forbidden effect observed: more than 1 crm.subscription_cancelled effect(s) observed within 10m |
| Change a shipping address | success | PASS | PASS | all 3 expected effects observed within 10m; preconditions, limits and prohibitions respected |
| Approve a discount | success | FAIL | FAIL | forbidden effect observed: more than 1 orders.discount_applied effect(s) observed within 10m |
| Close a support case | success | FAIL | FAIL | forbidden effect observed: more than 1 tickets.ticket_closed effect(s) observed within 10m |
| Send a payout | success | FAIL | FAIL | forbidden effect observed: more than 1 payments.payout_created effect(s) observed within 10m |
| Deploy a service | success | FAIL | FAIL | forbidden effect observed: more than 1 platform.deploy_created effect(s) observed within 10m |
| Place a trade | success | FAIL | FAIL | forbidden effect observed: more than 1 broker.order_created effect(s) observed within 10m |
Refund a customer
Scenario refund/lost_response_retry: timeout_after_success + retry without idempotency.
What the agent reported
The agent reported success. Its calls, and what it saw come back:
- GET /tickets/tick_1200
- GET /orders/order_1001200
- POST /payments/refundstimeoutinjected: lost_response
- POST /payments/refunds200
- POST /orders/order_1001/status200
- POST /crm/notes200
- POST /tickets/notify200
What the systems of record showed
Records written after the action:
- payments.refund_created on order_1001amount=180, currency=USD
- payments.refund_created on order_1001amount=180, currency=USD
- orders.order_status_changed on order_1001status=refunded
- crm.note_added on order_1001author=lab-agent, text=Refunded 180.0 USD per customer request
- tickets.customer_notified on order_1001channel=email
The oracle, reading the databases directly: FAIL
duplicate refund
The verdict and why
FAIL forbidden effect observed: more than 1 payments.refund_created effect(s) observed within 10m
Outcome PASS, policy FAIL; the oracle reads outcome PASS, policy FAIL.
- forbidden effectmore than 1 payments.refund_created effect(s) observed within 10m
Three models on the same injected failure
From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.
- AnthropicAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
- OpenAIAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
- GoogleAgent reported failure · trace-only judge saidFAIL· truth FAIL · CommitLayer FAIL
Cancel a subscription
Scenario cancel/lost_response_retry: timeout_after_success + retry without idempotency.
What the agent reported
The agent reported success. Its calls, and what it saw come back:
- GET /crm/subscriptions/sub_1200
- POST /crm/subscriptions/sub_1/canceltimeoutinjected: lost_response
- POST /crm/subscriptions/sub_1/cancel200
- POST /payments/billing/stop200
- POST /tickets/notify200
What the systems of record showed
Records written after the action:
- crm.subscription_cancelled on sub_1plan=pro
- crm.subscription_cancelled on sub_1plan=pro
- payments.billing_stopped on sub_1
- tickets.customer_notified on sub_1channel=email
The oracle, reading the databases directly: FAIL
duplicate cancellation
The verdict and why
FAIL forbidden effect observed: more than 1 crm.subscription_cancelled effect(s) observed within 10m
Outcome PASS, policy FAIL; the oracle reads outcome PASS, policy FAIL.
- forbidden effectmore than 1 crm.subscription_cancelled effect(s) observed within 10m
Three models on the same injected failure
From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.
- AnthropicAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
- OpenAIAgent reported success · trace-only judge saidFAIL· truth FAIL · CommitLayer FAIL
- GoogleAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
Change a shipping address
Scenario address/lost_response_retry: timeout_after_success + retry without idempotency.
What the agent reported
The agent reported success. Its calls, and what it saw come back:
- GET /orders/order_1001200
- POST /orders/order_1001/addresstimeoutinjected: lost_response
- POST /orders/order_1001/address200
- POST /crm/notes200
- POST /tickets/notify200
What the systems of record showed
Records written after the action:
- orders.address_updated on order_1001address=22 New Street, Lyon
- orders.address_updated on order_1001address=22 New Street, Lyon
- crm.note_added on order_1001author=lab-agent, text=Shipping address changed to 22 New Street, Lyon
- tickets.customer_notified on order_1001channel=email
The oracle, reading the databases directly: PASS
The verdict and why
PASS all 3 expected effects observed within 10m; preconditions, limits and prohibitions respected
Outcome PASS, policy PASS; the oracle reads outcome PASS, policy PASS.
Three models on the same injected failure
From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.
- AnthropicAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
- OpenAIAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
- GoogleAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
Approve a discount
Scenario discount/lost_response_retry: timeout_after_success + retry without idempotency.
What the agent reported
The agent reported success. Its calls, and what it saw come back:
- GET /orders/order_1004200
- POST /orders/order_1004/discounttimeoutinjected: lost_response
- POST /orders/order_1004/discount200
- POST /crm/notes200
What the systems of record showed
Records written after the action:
- orders.discount_applied on order_1004percent=15
- orders.discount_applied on order_1004percent=15
- crm.note_added on order_1004author=lab-agent, text=Discount 15.0% approved
The oracle, reading the databases directly: FAIL
duplicate discount
The verdict and why
FAIL forbidden effect observed: more than 1 orders.discount_applied effect(s) observed within 10m
Outcome PASS, policy FAIL; the oracle reads outcome PASS, policy FAIL.
- forbidden effectmore than 1 orders.discount_applied effect(s) observed within 10m
Three models on the same injected failure
From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.
- AnthropicAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
- OpenAIAgent reported success · trace-only judge saidFAIL· truth PASS · CommitLayer PASS
- GoogleAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
Close a support case
Scenario close/lost_response_retry: timeout_after_success + retry without idempotency.
What the agent reported
The agent reported success. Its calls, and what it saw come back:
- GET /tickets/tick_1200
- POST /tickets/tick_1/closetimeoutinjected: lost_response
- POST /tickets/tick_1/close200
- POST /crm/notes200
- POST /tickets/notify200
What the systems of record showed
Records written after the action:
- tickets.ticket_closed on tick_1order_id=order_1001
- tickets.ticket_closed on tick_1order_id=order_1001
- crm.note_added on tick_1author=lab-agent, text=Case closed: answered
- tickets.customer_notified on tick_1channel=email, order_id=order_1001
The oracle, reading the databases directly: FAIL
duplicate close
The verdict and why
FAIL forbidden effect observed: more than 1 tickets.ticket_closed effect(s) observed within 10m
Outcome PASS, policy FAIL; the oracle reads outcome PASS, policy FAIL.
- forbidden effectmore than 1 tickets.ticket_closed effect(s) observed within 10m
Three models on the same injected failure
From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.
- AnthropicAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
- OpenAIAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
- GoogleAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
Send a payout
Scenario payout/lost_response_retry: timeout_after_success + retry without idempotency.
What the agent reported
The agent reported success. Its calls, and what it saw come back:
- GET /ledger/accounts/acc_1200
- POST /payments/payoutstimeoutinjected: lost_response
- POST /payments/payouts200
- POST /ledger/accounts/acc_1/post200
- POST /crm/notes200
What the systems of record showed
Records written after the action:
- payments.payout_created on acc_1amount=900, currency=USD, payout_id=po_acc_1_1
- payments.payout_created on acc_1amount=900, currency=USD, payout_id=po_acc_1_2
- ledger.ledger_posted on acc_1amount=900, balance=9100
- crm.note_added on acc_1author=lab-agent, text=Payout of 900.0 USD sent
The oracle, reading the databases directly: FAIL
duplicate payout
The verdict and why
FAIL forbidden effect observed: more than 1 payments.payout_created effect(s) observed within 10m
Outcome PASS, policy FAIL; the oracle reads outcome PASS, policy FAIL.
- forbidden effectmore than 1 payments.payout_created effect(s) observed within 10m
Deploy a service
Scenario deploy/lost_response_retry: timeout_after_success + retry without idempotency.
What the agent reported
The agent reported success. Its calls, and what it saw come back:
- GET /platform/services/svc_api200
- POST /platform/services/svc_api/deploytimeoutinjected: lost_response
- POST /platform/services/svc_api/deploy200
- POST /platform/services/svc_api/healthcheck200
- POST /crm/notes200
What the systems of record showed
Records written after the action:
- platform.deploy_created on svc_apiversion=1.5.0, environment=production
- platform.deploy_created on svc_apiversion=1.5.0, environment=production
- platform.health_check on svc_apistatus=healthy, version=1.5.0
- crm.note_added on svc_apiauthor=lab-agent, text=Deployed 1.5.0 to production
The oracle, reading the databases directly: FAIL
duplicate deploy
The verdict and why
FAIL forbidden effect observed: more than 1 platform.deploy_created effect(s) observed within 10m
Outcome PASS, policy FAIL; the oracle reads outcome PASS, policy FAIL.
- forbidden effectmore than 1 platform.deploy_created effect(s) observed within 10m
Place a trade
Scenario trade/lost_response_retry: timeout_after_success + retry without idempotency.
What the agent reported
The agent reported success. Its calls, and what it saw come back:
- GET /broker/products/TSLA-USD200
- POST /broker/orderstimeoutinjected: lost_response
- POST /broker/orders200
- GET /broker/orders/ord_TSLA-USD_2200
- POST /crm/notes200
What the systems of record showed
Records written after the action:
- broker.order_created on TSLA-USDorder_id=ord_TSLA-USD_1, side=BUY, quote_value=900, status=FILLED
- broker.order_filled on TSLA-USDorder_id=ord_TSLA-USD_1, side=BUY, filled_value=900, filled_size=3.4351145, average_filled_price=262
- broker.order_created on TSLA-USDorder_id=ord_TSLA-USD_2, side=BUY, quote_value=900, status=FILLED
- broker.order_filled on TSLA-USDorder_id=ord_TSLA-USD_2, side=BUY, filled_value=900, filled_size=3.4351145, average_filled_price=262
- crm.note_added on TSLA-USDauthor=lab-agent, text=BUY 900.0 USD of TSLA-USD
The oracle, reading the databases directly: FAIL
duplicate order
The verdict and why
FAIL forbidden effect observed: more than 1 broker.order_created effect(s) observed within 10m
Outcome PASS, policy FAIL; the oracle reads outcome PASS, policy FAIL.
- forbidden effectmore than 1 broker.order_created effect(s) observed within 10m
Run it in the Failure Lab
The Failure Lab replays each run above: the agent's calls, the records the systems held and the verdict. No account is needed. The verdict comes from the production engine while you watch.
The sandbox in the app runs the exact scenario in your own workspace, as a verified action: