System unreachable
Injected failure: a required system is down at verification time. Extraction cannot reach the primary system; the verifier should say UNKNOWN, not PASS or FAIL.
Mode system_unreachable of the failure matrix, on all eight workflows. Each run below is the sandbox's scripted agent acting on the sandbox's fake systems of record under the injected failure, an oracle reading those systems' databases, and the verdict of engine 1.0.0 from the systems of record alone.
Last changed Markdown versionllms.txt
| Workflow | Agent reported | Oracle | CommitLayer | Why |
|---|---|---|---|---|
| Refund a customer | success | UNKNOWN | UNKNOWN | Required fact source not connected |
| Cancel a subscription | success | UNKNOWN | UNKNOWN | Required fact source not connected |
| Change a shipping address | success | UNKNOWN | UNKNOWN | Required fact source not connected |
| Approve a discount | success | UNKNOWN | UNKNOWN | Required fact source not connected |
| Close a support case | success | UNKNOWN | UNKNOWN | Required fact source not connected |
| Send a payout | success | UNKNOWN | UNKNOWN | Required fact source not connected |
| Deploy a service | success | UNKNOWN | UNKNOWN | Required fact source not connected |
| Place a trade | success | UNKNOWN | UNKNOWN | Required fact source not connected |
Refund a customer
Scenario refund/system_unreachable: a required system is down at verification time.
What the agent reported
The agent reported success. Its calls, and what it saw come back:
- GET /tickets/tick_1200
- GET /orders/order_1001200
- POST /payments/refunds200
- POST /orders/order_1001/status200
- POST /crm/notes200
- POST /tickets/notify200
What the systems of record showed
Records written after the action:
- orders.order_status_changed on order_1001status=refunded
- crm.note_added on order_1001author=lab-agent, text=Refunded 180.0 USD per customer request
- tickets.customer_notified on order_1001channel=email
Not reachable when the records were read: payments.
The oracle, reading the databases directly: UNKNOWN
required system unreachable
The verdict and why
UNKNOWN Required fact source not connected
Outcome UNKNOWN, policy UNKNOWN; the oracle reads outcome UNKNOWN, policy UNKNOWN.
- expected effectCould not be checked: payments was not reachablepayments.refund_created observed within 10m of the action
Three models on the same injected failure
From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.
- AnthropicAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN
- OpenAIAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN
- GoogleAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN
Cancel a subscription
Scenario cancel/system_unreachable: a required system is down at verification time.
What the agent reported
The agent reported success. Its calls, and what it saw come back:
- GET /crm/subscriptions/sub_1200
- POST /crm/subscriptions/sub_1/cancel200
- POST /payments/billing/stop200
- POST /tickets/notify200
What the systems of record showed
Records written after the action:
- payments.billing_stopped on sub_1
- tickets.customer_notified on sub_1channel=email
Not reachable when the records were read: crm.
The oracle, reading the databases directly: UNKNOWN
required system unreachable
The verdict and why
UNKNOWN Required fact source not connected
Outcome UNKNOWN, policy UNKNOWN; the oracle reads outcome UNKNOWN, policy UNKNOWN.
- preconditionCould not be checked: crm was not reachablelatest crm.subscription_status before the action had status=active
- expected effectCould not be checked: crm was not reachablecrm.subscription_cancelled observed within 10m of the action
Three models on the same injected failure
From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.
- AnthropicAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN
- OpenAIAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN
- GoogleAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN
Change a shipping address
Scenario address/system_unreachable: a required system is down at verification time.
What the agent reported
The agent reported success. Its calls, and what it saw come back:
- GET /orders/order_1001200
- POST /orders/order_1001/address200
- POST /crm/notes200
- POST /tickets/notify200
What the systems of record showed
Records written after the action:
- crm.note_added on order_1001author=lab-agent, text=Shipping address changed to 22 New Street, Lyon
- tickets.customer_notified on order_1001channel=email
Not reachable when the records were read: orders.
The oracle, reading the databases directly: UNKNOWN
required system unreachable
The verdict and why
UNKNOWN Required fact source not connected
Outcome UNKNOWN, policy UNKNOWN; the oracle reads outcome UNKNOWN, policy UNKNOWN.
- preconditionCould not be checked: orders was not reachablelatest orders.order_status before the action had status=paid
- expected effectCould not be checked: orders was not reachableorders.address_updated observed within 10m of the action
Three models on the same injected failure
From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.
- AnthropicAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN
- OpenAIAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN
- GoogleAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN
Approve a discount
Scenario discount/system_unreachable: a required system is down at verification time.
What the agent reported
The agent reported success. Its calls, and what it saw come back:
- GET /orders/order_1004200
- POST /orders/order_1004/discount200
- POST /crm/notes200
What the systems of record showed
Records written after the action:
- crm.note_added on order_1004author=lab-agent, text=Discount 15.0% approved
Not reachable when the records were read: orders.
The oracle, reading the databases directly: UNKNOWN
required system unreachable
The verdict and why
UNKNOWN Required fact source not connected
Outcome UNKNOWN, policy UNKNOWN; the oracle reads outcome UNKNOWN, policy UNKNOWN.
- preconditionCould not be checked: orders was not reachablelatest orders.order_status before the action had status=pending_payment
- expected effectCould not be checked: orders was not reachableorders.discount_applied observed within 10m of the action
Three models on the same injected failure
From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.
- AnthropicAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN
- OpenAIAgent reported success · trace-only judge saidFAIL· truth UNKNOWN · CommitLayer UNKNOWN
- GoogleAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN
Close a support case
Scenario close/system_unreachable: a required system is down at verification time.
What the agent reported
The agent reported success. Its calls, and what it saw come back:
- GET /tickets/tick_1200
- POST /tickets/tick_1/close200
- POST /crm/notes200
- POST /tickets/notify200
What the systems of record showed
Records written after the action:
- crm.note_added on tick_1author=lab-agent, text=Case closed: answered
Not reachable when the records were read: tickets.
The oracle, reading the databases directly: UNKNOWN
required system unreachable
The verdict and why
UNKNOWN Required fact source not connected
Outcome UNKNOWN, policy UNKNOWN; the oracle reads outcome UNKNOWN, policy UNKNOWN.
- preconditionCould not be checked: tickets was not reachablelatest tickets.ticket_status before the action had status=open
- expected effectCould not be checked: tickets was not reachabletickets.ticket_closed observed within 10m of the action
- expected effectCould not be checked: tickets was not reachabletickets.customer_notified observed within 10m of the action
Three models on the same injected failure
From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.
- AnthropicAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN
- OpenAIAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN
- GoogleAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN
Send a payout
Scenario payout/system_unreachable: a required system is down at verification time.
What the agent reported
The agent reported success. Its calls, and what it saw come back:
- GET /ledger/accounts/acc_1200
- POST /payments/payouts200
- POST /ledger/accounts/acc_1/post200
- POST /crm/notes200
What the systems of record showed
Records written after the action:
- ledger.ledger_posted on acc_1amount=900, balance=9100
- crm.note_added on acc_1author=lab-agent, text=Payout of 900.0 USD sent
Not reachable when the records were read: payments.
The oracle, reading the databases directly: UNKNOWN
required system unreachable
The verdict and why
UNKNOWN Required fact source not connected
Outcome UNKNOWN, policy UNKNOWN; the oracle reads outcome UNKNOWN, policy UNKNOWN.
- expected effectCould not be checked: payments was not reachablepayments.payout_created observed within 10m of the action
Deploy a service
Scenario deploy/system_unreachable: a required system is down at verification time.
What the agent reported
The agent reported success. Its calls, and what it saw come back:
- GET /platform/services/svc_api200
- POST /platform/services/svc_api/deploy200
- POST /platform/services/svc_api/healthcheck200
- POST /crm/notes200
What the systems of record showed
Records written after the action:
- crm.note_added on svc_apiauthor=lab-agent, text=Deployed 1.5.0 to production
Not reachable when the records were read: platform.
The oracle, reading the databases directly: UNKNOWN
required system unreachable
The verdict and why
UNKNOWN Required fact source not connected
Outcome UNKNOWN, policy UNKNOWN; the oracle reads outcome UNKNOWN, policy UNKNOWN.
- preconditionCould not be checked: platform was not reachablelatest platform.service_health before the action had status=healthy
- expected effectCould not be checked: platform was not reachableplatform.deploy_created observed within 10m of the action
- expected effectCould not be checked: platform was not reachableplatform.health_check observed within 10m of the action where status=healthy
Place a trade
Scenario trade/system_unreachable: a required system is down at verification time.
What the agent reported
The agent reported success. Its calls, and what it saw come back:
- GET /broker/products/TSLA-USD200
- POST /broker/orders200
- GET /broker/orders/ord_TSLA-USD_1200
- POST /crm/notes200
What the systems of record showed
Records written after the action:
- crm.note_added on TSLA-USDauthor=lab-agent, text=BUY 900.0 USD of TSLA-USD
Not reachable when the records were read: broker.
The oracle, reading the databases directly: UNKNOWN
required system unreachable
The verdict and why
UNKNOWN Required fact source not connected
Outcome UNKNOWN, policy UNKNOWN; the oracle reads outcome UNKNOWN, policy UNKNOWN.
- preconditionCould not be checked: broker was not reachablelatest broker.product_status before the action had status=tradable
- expected effectCould not be checked: broker was not reachablebroker.order_created observed within 10m of the action
- expected effectCould not be checked: broker was not reachablebroker.order_filled observed within 10m of the action
Run it in the Failure Lab
The Failure Lab replays each run above: the agent's calls, the records the systems held and the verdict. No account is needed. The verdict it shows is the one recorded with the run: the run turns on a system that could not be read, and the dry run the lab sends cannot say a system was down.
The sandbox in the app runs the exact scenario in your own workspace, as a verified action: