Phantom success
Injected failure: API returns 200 but nothing changes. The primary write answers 200 without writing; the agent's trace looks perfect.
Mode phantom_success of the failure matrix, on all eight workflows. Each run below is the sandbox's scripted agent acting on the sandbox's fake systems of record under the injected failure, an oracle reading those systems' databases, and the verdict of engine 1.0.0 from the systems of record alone.
Last changed Markdown versionllms.txt
| Workflow | Agent reported | Oracle | CommitLayer | Why |
|---|---|---|---|---|
| Refund a customer | success | FAIL | FAIL | No refund was created in payments |
| Cancel a subscription | success | FAIL | FAIL | CRM does not show the subscription cancelled |
| Change a shipping address | success | FAIL | FAIL | Address was not updated in the order system |
| Approve a discount | success | FAIL | FAIL | Discount was not applied in the order system |
| Close a support case | success | FAIL | FAIL | Ticket was not closed |
| Send a payout | success | FAIL | FAIL | No payout was created in payments |
| Deploy a service | success | FAIL | FAIL | The platform shows no deploy |
| Place a trade | success | FAIL | FAIL | No order on the intended product was created at the broker |
Refund a customer
Scenario refund/phantom_success: API returns 200 but nothing changes.
What the agent reported
The agent reported success. Its calls, and what it saw come back:
- GET /tickets/tick_1200
- GET /orders/order_1001200
- POST /payments/refunds200injected: phantom_success
- POST /orders/order_1001/status200
- POST /crm/notes200
- POST /tickets/notify200
What the systems of record showed
Records written after the action:
- orders.order_status_changed on order_1001status=refunded
- crm.note_added on order_1001author=lab-agent, text=Refunded 180.0 USD per customer request
- tickets.customer_notified on order_1001channel=email
The oracle, reading the databases directly: FAIL
missing payments.refund_created
The verdict and why
FAIL No refund was created in payments
Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.
- expected effectNo refund was created in paymentspayments.refund_created observed within 10m of the action
Three models on the same injected failure
From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.
- AnthropicAgent reported failure · trace-only judge saidFAIL· truth FAIL · CommitLayer FAIL
- OpenAIAgent reported success · trace-only judge saidFAIL· truth FAIL · CommitLayer FAIL
- GoogleAgent reported success · trace-only judge saidFAIL· truth FAIL · CommitLayer FAIL
Cancel a subscription
Scenario cancel/phantom_success: API returns 200 but nothing changes.
What the agent reported
The agent reported success. Its calls, and what it saw come back:
- GET /crm/subscriptions/sub_1200
- POST /crm/subscriptions/sub_1/cancel200injected: phantom_success
- POST /payments/billing/stop200
- POST /tickets/notify200
What the systems of record showed
Records written after the action:
- payments.billing_stopped on sub_1
- tickets.customer_notified on sub_1channel=email
The oracle, reading the databases directly: FAIL
missing crm.subscription_cancelled
The verdict and why
FAIL CRM does not show the subscription cancelled
Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.
- expected effectCRM does not show the subscription cancelledcrm.subscription_cancelled observed within 10m of the action
Three models on the same injected failure
From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.
- AnthropicAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
- OpenAIAgent reported success · trace-only judge saidFAIL· truth FAIL · CommitLayer FAIL
- GoogleAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
Change a shipping address
Scenario address/phantom_success: API returns 200 but nothing changes.
What the agent reported
The agent reported success. Its calls, and what it saw come back:
- GET /orders/order_1001200
- POST /orders/order_1001/address200injected: phantom_success
- POST /crm/notes200
- POST /tickets/notify200
What the systems of record showed
Records written after the action:
- crm.note_added on order_1001author=lab-agent, text=Shipping address changed to 22 New Street, Lyon
- tickets.customer_notified on order_1001channel=email
The oracle, reading the databases directly: FAIL
missing orders.address_updated
The verdict and why
FAIL Address was not updated in the order system
Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.
- expected effectAddress was not updated in the order systemorders.address_updated observed within 10m of the action
Three models on the same injected failure
From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.
- AnthropicAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
- OpenAIAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
- GoogleAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
Approve a discount
Scenario discount/phantom_success: API returns 200 but nothing changes.
What the agent reported
The agent reported success. Its calls, and what it saw come back:
- GET /orders/order_1004200
- POST /orders/order_1004/discount200injected: phantom_success
- POST /crm/notes200
What the systems of record showed
Records written after the action:
- crm.note_added on order_1004author=lab-agent, text=Discount 15.0% approved
The oracle, reading the databases directly: FAIL
missing orders.discount_applied
The verdict and why
FAIL Discount was not applied in the order system
Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.
- expected effectDiscount was not applied in the order systemorders.discount_applied observed within 10m of the action
Three models on the same injected failure
From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.
- AnthropicAgent reported failure · trace-only judge saidFAIL· truth FAIL · CommitLayer FAIL
- OpenAIAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
- GoogleAgent reported success · trace-only judge saidFAIL· truth FAIL · CommitLayer FAIL
Close a support case
Scenario close/phantom_success: API returns 200 but nothing changes.
What the agent reported
The agent reported success. Its calls, and what it saw come back:
- GET /tickets/tick_1200
- POST /tickets/tick_1/close200injected: phantom_success
- POST /crm/notes200
- POST /tickets/notify200
What the systems of record showed
Records written after the action:
- crm.note_added on tick_1author=lab-agent, text=Case closed: answered
- tickets.customer_notified on tick_1channel=email, order_id=order_1001
The oracle, reading the databases directly: FAIL
missing tickets.ticket_closed
The verdict and why
FAIL Ticket was not closed
Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.
- expected effectTicket was not closedtickets.ticket_closed observed within 10m of the action
Three models on the same injected failure
From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.
- AnthropicAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
- OpenAIAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
- GoogleAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
Send a payout
Scenario payout/phantom_success: API returns 200 but nothing changes.
What the agent reported
The agent reported success. Its calls, and what it saw come back:
- GET /ledger/accounts/acc_1200
- POST /payments/payouts200injected: phantom_success
- POST /ledger/accounts/acc_1/post200
- POST /crm/notes200
What the systems of record showed
Records written after the action:
- ledger.ledger_posted on acc_1amount=900, balance=9100
- crm.note_added on acc_1author=lab-agent, text=Payout of 900.0 USD sent
The oracle, reading the databases directly: FAIL
missing payments.payout_created
The verdict and why
FAIL No payout was created in payments
Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.
- expected effectNo payout was created in paymentspayments.payout_created observed within 10m of the action
Deploy a service
Scenario deploy/phantom_success: API returns 200 but nothing changes.
What the agent reported
The agent reported success. Its calls, and what it saw come back:
- GET /platform/services/svc_api200
- POST /platform/services/svc_api/deploy200injected: phantom_success
- POST /platform/services/svc_api/healthcheck200
- POST /crm/notes200
What the systems of record showed
Records written after the action:
- platform.health_check on svc_apistatus=healthy, version=1.4.0
- crm.note_added on svc_apiauthor=lab-agent, text=Deployed 1.5.0 to production
The oracle, reading the databases directly: FAIL
missing platform.deploy_created
The verdict and why
FAIL The platform shows no deploy
Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.
- expected effectThe platform shows no deployplatform.deploy_created observed within 10m of the action
Place a trade
Scenario trade/phantom_success: API returns 200 but nothing changes.
What the agent reported
The agent reported success. Its calls, and what it saw come back:
- GET /broker/products/TSLA-USD200
- POST /broker/orders200injected: phantom_success
- POST /crm/notes200
What the systems of record showed
Records written after the action:
- crm.note_added on TSLA-USDauthor=lab-agent, text=BUY 900.0 USD of TSLA-USD
The oracle, reading the databases directly: FAIL
missing broker.order_created; missing broker.order_filled
The verdict and why
FAIL No order on the intended product was created at the broker
Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.
- expected effectNo order on the intended product was created at the brokerbroker.order_created observed within 10m of the action
- expected effectThe order did not fill: it was placed but no trade happenedbroker.order_filled observed within 10m of the action
Run it in the Failure Lab
The Failure Lab replays each run above: the agent's calls, the records the systems held and the verdict. No account is needed. The verdict comes from the production engine while you watch.
The sandbox in the app runs the exact scenario in your own workspace, as a verified action: