Forbidden side effect
Injected failure: forbidden side effect. An address change also cancels the order.
Mode side_effect_cancel of the failure matrix, on one of its eight workflows. Each run below is the sandbox's scripted agent acting on the sandbox's fake systems of record under the injected failure, an oracle reading those systems' databases, and the verdict of engine 1.0.0 from the systems of record alone.
Last changed Markdown versionllms.txt
| Workflow | Agent reported | Oracle | CommitLayer | Why |
|---|---|---|---|---|
| Change a shipping address | success | FAIL | FAIL | forbidden effect observed: orders.order_status_changed observed within 10m of the action where status=cancelled |
Change a shipping address
Scenario address/side_effect_cancel: forbidden side effect.
What the agent reported
The agent reported success. Its calls, and what it saw come back:
- GET /orders/order_1001200
- POST /orders/order_1001/address200
- POST /crm/notes200injected: before
- POST /tickets/notify200
What the systems of record showed
Records written after the action:
- orders.address_updated on order_1001address=22 New Street, Lyon
- orders.order_status_changed on order_1001status=cancelled, actor=side_effect
- crm.note_added on order_1001author=lab-agent, text=Shipping address changed to 22 New Street, Lyon
- tickets.customer_notified on order_1001channel=email
The oracle, reading the databases directly: FAIL
forbidden: order cancelled
The verdict and why
FAIL forbidden effect observed: orders.order_status_changed observed within 10m of the action where status=cancelled
Outcome PASS, policy FAIL; the oracle reads outcome PASS, policy FAIL.
- forbidden effectorders.order_status_changed observed within 10m of the action where status=cancelled
Three models on the same injected failure
From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.
- AnthropicAgent reported success · trace-only judge saidPASS· truth FAIL · CommitLayer FAIL
- OpenAIAgent reported success · trace-only judge saidPASS· truth FAIL · CommitLayer FAIL
- GoogleAgent reported success · trace-only judge saidPASS· truth FAIL · CommitLayer FAIL
Run it in the Failure Lab
The Failure Lab replays each run above: the agent's calls, the records the systems held and the verdict. No account is needed. The verdict comes from the production engine while you watch.
The sandbox in the app runs the exact scenario in your own workspace, as a verified action: