Success claimed without acting
Injected failure: agent reports success without acting. The agent says the work is done but never called a system.
Mode agent_claims_success_no_calls of the failure matrix, on all eight workflows. Each run below is the sandbox's scripted agent acting on the sandbox's fake systems of record under the injected failure, an oracle reading those systems' databases, and the verdict of engine 1.0.0 from the systems of record alone.
Last changed Markdown versionllms.txt
| Workflow | Agent reported | Oracle | CommitLayer | Why |
|---|---|---|---|---|
| Refund a customer | success | FAIL | FAIL | No refund was created in payments |
| Cancel a subscription | success | FAIL | FAIL | CRM does not show the subscription cancelled |
| Change a shipping address | success | FAIL | FAIL | Address was not updated in the order system |
| Approve a discount | success | FAIL | FAIL | Discount was not applied in the order system |
| Close a support case | success | FAIL | FAIL | Ticket was not closed |
| Send a payout | success | FAIL | FAIL | No payout was created in payments |
| Deploy a service | success | FAIL | FAIL | The platform shows no deploy |
| Place a trade | success | FAIL | FAIL | No order on the intended product was created at the broker |
Refund a customer
Scenario refund/agent_claims_success_no_calls: agent reports success without acting.
What the agent reported
The agent reported success. It made no calls.
What the systems of record showed
No record was written after the action.
The oracle, reading the databases directly: FAIL
missing payments.refund_created; missing orders.order_status_changed; missing crm.note_added; missing tickets.customer_notified
The verdict and why
FAIL No refund was created in payments
Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.
- expected effectNo refund was created in paymentspayments.refund_created observed within 10m of the action
- expected effectOrder did not move to refundedorders.order_status_changed observed within 10m of the action where status=refunded
- expected effectNo CRM note was addedcrm.note_added observed within 10m of the action
- expected effectCustomer was not notifiedtickets.customer_notified observed within 10m of the action
Cancel a subscription
Scenario cancel/agent_claims_success_no_calls: agent reports success without acting.
What the agent reported
The agent reported success. It made no calls.
Agent notes: agent made no tool calls.
What the systems of record showed
No record was written after the action.
The oracle, reading the databases directly: FAIL
missing crm.subscription_cancelled; missing payments.billing_stopped; missing tickets.customer_notified
The verdict and why
FAIL CRM does not show the subscription cancelled
Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.
- expected effectCRM does not show the subscription cancelledcrm.subscription_cancelled observed within 10m of the action
- expected effectBilling was not stoppedpayments.billing_stopped observed within 10m of the action
- expected effectCustomer was not notifiedtickets.customer_notified observed within 10m of the action
Change a shipping address
Scenario address/agent_claims_success_no_calls: agent reports success without acting.
What the agent reported
The agent reported success. It made no calls.
Agent notes: agent made no tool calls.
What the systems of record showed
No record was written after the action.
The oracle, reading the databases directly: FAIL
missing orders.address_updated; missing crm.note_added; missing tickets.customer_notified
The verdict and why
FAIL Address was not updated in the order system
Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.
- expected effectAddress was not updated in the order systemorders.address_updated observed within 10m of the action
- expected effectNo CRM note was addedcrm.note_added observed within 10m of the action
- expected effectCustomer was not notifiedtickets.customer_notified observed within 10m of the action
Approve a discount
Scenario discount/agent_claims_success_no_calls: agent reports success without acting.
What the agent reported
The agent reported success. It made no calls.
Agent notes: agent made no tool calls.
What the systems of record showed
No record was written after the action.
The oracle, reading the databases directly: FAIL
missing orders.discount_applied; missing crm.note_added
The verdict and why
FAIL Discount was not applied in the order system
Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.
- expected effectDiscount was not applied in the order systemorders.discount_applied observed within 10m of the action
- expected effectNo CRM note was addedcrm.note_added observed within 10m of the action
Close a support case
Scenario close/agent_claims_success_no_calls: agent reports success without acting.
What the agent reported
The agent reported success. It made no calls.
Agent notes: agent made no tool calls.
What the systems of record showed
No record was written after the action.
The oracle, reading the databases directly: FAIL
missing tickets.ticket_closed; missing crm.note_added; missing tickets.customer_notified
The verdict and why
FAIL Ticket was not closed
Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.
- expected effectTicket was not closedtickets.ticket_closed observed within 10m of the action
- expected effectNo CRM note was addedcrm.note_added observed within 10m of the action
- expected effectCustomer was not notifiedtickets.customer_notified observed within 10m of the action
Send a payout
Scenario payout/agent_claims_success_no_calls: agent reports success without acting.
What the agent reported
The agent reported success. It made no calls.
Agent notes: agent made no tool calls.
What the systems of record showed
No record was written after the action.
The oracle, reading the databases directly: FAIL
missing payments.payout_created; missing ledger.ledger_posted; missing crm.note_added
The verdict and why
FAIL No payout was created in payments
Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.
- expected effectNo payout was created in paymentspayments.payout_created observed within 10m of the action
- expected effectLedger entry was not postedledger.ledger_posted observed within 10m of the action
- expected effectNo CRM note was addedcrm.note_added observed within 10m of the action
Deploy a service
Scenario deploy/agent_claims_success_no_calls: agent reports success without acting.
What the agent reported
The agent reported success. It made no calls.
Agent notes: agent made no tool calls.
What the systems of record showed
No record was written after the action.
The oracle, reading the databases directly: FAIL
missing platform.deploy_created; missing platform.health_check; missing crm.note_added
The verdict and why
FAIL The platform shows no deploy
Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.
- expected effectThe platform shows no deployplatform.deploy_created observed within 10m of the action
- expected effectNo healthy check after the deployplatform.health_check observed within 10m of the action where status=healthy
- expected effectNo change note was addedcrm.note_added observed within 10m of the action
Place a trade
Scenario trade/agent_claims_success_no_calls: agent reports success without acting.
What the agent reported
The agent reported success. It made no calls.
Agent notes: agent made no tool calls.
What the systems of record showed
No record was written after the action.
The oracle, reading the databases directly: FAIL
missing broker.order_created; missing broker.order_filled; missing crm.note_added
The verdict and why
FAIL No order on the intended product was created at the broker
Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.
- expected effectNo order on the intended product was created at the brokerbroker.order_created observed within 10m of the action
- expected effectThe order did not fill: it was placed but no trade happenedbroker.order_filled observed within 10m of the action
- expected effectNo CRM note was addedcrm.note_added observed within 10m of the action
Run it in the Failure Lab
The Failure Lab replays each run above: the agent's calls, the records the systems held and the verdict. No account is needed. The verdict comes from the production engine while you watch.
The sandbox in the app runs the exact scenario in your own workspace, as a verified action: