Skip to content
All failure modes
Failure mode

System unreachable

Injected failure: a required system is down at verification time. Extraction cannot reach the primary system; the verifier should say UNKNOWN, not PASS or FAIL.

Mode system_unreachable of the failure matrix, on all eight workflows. Each run below is the sandbox's scripted agent acting on the sandbox's fake systems of record under the injected failure, an oracle reading those systems' databases, and the verdict of engine 1.0.0 from the systems of record alone.

Synthetic demo data.

Last changed Markdown versionllms.txt

WorkflowAgent reportedOracleCommitLayerWhy
Refund a customersuccessUNKNOWNUNKNOWNRequired fact source not connected
Cancel a subscriptionsuccessUNKNOWNUNKNOWNRequired fact source not connected
Change a shipping addresssuccessUNKNOWNUNKNOWNRequired fact source not connected
Approve a discountsuccessUNKNOWNUNKNOWNRequired fact source not connected
Close a support casesuccessUNKNOWNUNKNOWNRequired fact source not connected
Send a payoutsuccessUNKNOWNUNKNOWNRequired fact source not connected
Deploy a servicesuccessUNKNOWNUNKNOWNRequired fact source not connected
Place a tradesuccessUNKNOWNUNKNOWNRequired fact source not connected

Refund a customer

Scenario refund/system_unreachable: a required system is down at verification time.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /tickets/tick_1200
  • GET /orders/order_1001200
  • POST /payments/refunds200
  • POST /orders/order_1001/status200
  • POST /crm/notes200
  • POST /tickets/notify200

What the systems of record showed

Records written after the action:

  • orders.order_status_changed on order_1001
    status=refunded
  • crm.note_added on order_1001
    author=lab-agent, text=Refunded 180.0 USD per customer request
  • tickets.customer_notified on order_1001
    channel=email

Not reachable when the records were read: payments.

The oracle, reading the databases directly: UNKNOWN

required system unreachable

The verdict and why

UNKNOWN Required fact source not connected

Outcome UNKNOWN, policy UNKNOWN; the oracle reads outcome UNKNOWN, policy UNKNOWN.

  • expected effect
    Could not be checked: payments was not reachable
    payments.refund_created observed within 10m of the action

Three models on the same injected failure

From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.

  • AnthropicAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN
  • OpenAIAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN
  • GoogleAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN

Cancel a subscription

Scenario cancel/system_unreachable: a required system is down at verification time.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /crm/subscriptions/sub_1200
  • POST /crm/subscriptions/sub_1/cancel200
  • POST /payments/billing/stop200
  • POST /tickets/notify200

What the systems of record showed

Records written after the action:

  • payments.billing_stopped on sub_1
  • tickets.customer_notified on sub_1
    channel=email

Not reachable when the records were read: crm.

The oracle, reading the databases directly: UNKNOWN

required system unreachable

The verdict and why

UNKNOWN Required fact source not connected

Outcome UNKNOWN, policy UNKNOWN; the oracle reads outcome UNKNOWN, policy UNKNOWN.

  • precondition
    Could not be checked: crm was not reachable
    latest crm.subscription_status before the action had status=active
  • expected effect
    Could not be checked: crm was not reachable
    crm.subscription_cancelled observed within 10m of the action

Three models on the same injected failure

From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.

  • AnthropicAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN
  • OpenAIAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN
  • GoogleAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN

Change a shipping address

Scenario address/system_unreachable: a required system is down at verification time.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /orders/order_1001200
  • POST /orders/order_1001/address200
  • POST /crm/notes200
  • POST /tickets/notify200

What the systems of record showed

Records written after the action:

  • crm.note_added on order_1001
    author=lab-agent, text=Shipping address changed to 22 New Street, Lyon
  • tickets.customer_notified on order_1001
    channel=email

Not reachable when the records were read: orders.

The oracle, reading the databases directly: UNKNOWN

required system unreachable

The verdict and why

UNKNOWN Required fact source not connected

Outcome UNKNOWN, policy UNKNOWN; the oracle reads outcome UNKNOWN, policy UNKNOWN.

  • precondition
    Could not be checked: orders was not reachable
    latest orders.order_status before the action had status=paid
  • expected effect
    Could not be checked: orders was not reachable
    orders.address_updated observed within 10m of the action

Three models on the same injected failure

From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.

  • AnthropicAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN
  • OpenAIAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN
  • GoogleAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN

Approve a discount

Scenario discount/system_unreachable: a required system is down at verification time.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /orders/order_1004200
  • POST /orders/order_1004/discount200
  • POST /crm/notes200

What the systems of record showed

Records written after the action:

  • crm.note_added on order_1004
    author=lab-agent, text=Discount 15.0% approved

Not reachable when the records were read: orders.

The oracle, reading the databases directly: UNKNOWN

required system unreachable

The verdict and why

UNKNOWN Required fact source not connected

Outcome UNKNOWN, policy UNKNOWN; the oracle reads outcome UNKNOWN, policy UNKNOWN.

  • precondition
    Could not be checked: orders was not reachable
    latest orders.order_status before the action had status=pending_payment
  • expected effect
    Could not be checked: orders was not reachable
    orders.discount_applied observed within 10m of the action

Three models on the same injected failure

From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.

  • AnthropicAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN
  • OpenAIAgent reported success · trace-only judge saidFAIL· truth UNKNOWN · CommitLayer UNKNOWN
  • GoogleAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN

Close a support case

Scenario close/system_unreachable: a required system is down at verification time.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /tickets/tick_1200
  • POST /tickets/tick_1/close200
  • POST /crm/notes200
  • POST /tickets/notify200

What the systems of record showed

Records written after the action:

  • crm.note_added on tick_1
    author=lab-agent, text=Case closed: answered

Not reachable when the records were read: tickets.

The oracle, reading the databases directly: UNKNOWN

required system unreachable

The verdict and why

UNKNOWN Required fact source not connected

Outcome UNKNOWN, policy UNKNOWN; the oracle reads outcome UNKNOWN, policy UNKNOWN.

  • precondition
    Could not be checked: tickets was not reachable
    latest tickets.ticket_status before the action had status=open
  • expected effect
    Could not be checked: tickets was not reachable
    tickets.ticket_closed observed within 10m of the action
  • expected effect
    Could not be checked: tickets was not reachable
    tickets.customer_notified observed within 10m of the action

Three models on the same injected failure

From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.

  • AnthropicAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN
  • OpenAIAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN
  • GoogleAgent reported success · trace-only judge saidPASS· truth UNKNOWN · CommitLayer UNKNOWN

Send a payout

Scenario payout/system_unreachable: a required system is down at verification time.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /ledger/accounts/acc_1200
  • POST /payments/payouts200
  • POST /ledger/accounts/acc_1/post200
  • POST /crm/notes200

What the systems of record showed

Records written after the action:

  • ledger.ledger_posted on acc_1
    amount=900, balance=9100
  • crm.note_added on acc_1
    author=lab-agent, text=Payout of 900.0 USD sent

Not reachable when the records were read: payments.

The oracle, reading the databases directly: UNKNOWN

required system unreachable

The verdict and why

UNKNOWN Required fact source not connected

Outcome UNKNOWN, policy UNKNOWN; the oracle reads outcome UNKNOWN, policy UNKNOWN.

  • expected effect
    Could not be checked: payments was not reachable
    payments.payout_created observed within 10m of the action

Deploy a service

Scenario deploy/system_unreachable: a required system is down at verification time.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /platform/services/svc_api200
  • POST /platform/services/svc_api/deploy200
  • POST /platform/services/svc_api/healthcheck200
  • POST /crm/notes200

What the systems of record showed

Records written after the action:

  • crm.note_added on svc_api
    author=lab-agent, text=Deployed 1.5.0 to production

Not reachable when the records were read: platform.

The oracle, reading the databases directly: UNKNOWN

required system unreachable

The verdict and why

UNKNOWN Required fact source not connected

Outcome UNKNOWN, policy UNKNOWN; the oracle reads outcome UNKNOWN, policy UNKNOWN.

  • precondition
    Could not be checked: platform was not reachable
    latest platform.service_health before the action had status=healthy
  • expected effect
    Could not be checked: platform was not reachable
    platform.deploy_created observed within 10m of the action
  • expected effect
    Could not be checked: platform was not reachable
    platform.health_check observed within 10m of the action where status=healthy

Place a trade

Scenario trade/system_unreachable: a required system is down at verification time.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /broker/products/TSLA-USD200
  • POST /broker/orders200
  • GET /broker/orders/ord_TSLA-USD_1200
  • POST /crm/notes200

What the systems of record showed

Records written after the action:

  • crm.note_added on TSLA-USD
    author=lab-agent, text=BUY 900.0 USD of TSLA-USD

Not reachable when the records were read: broker.

The oracle, reading the databases directly: UNKNOWN

required system unreachable

The verdict and why

UNKNOWN Required fact source not connected

Outcome UNKNOWN, policy UNKNOWN; the oracle reads outcome UNKNOWN, policy UNKNOWN.

  • precondition
    Could not be checked: broker was not reachable
    latest broker.product_status before the action had status=tradable
  • expected effect
    Could not be checked: broker was not reachable
    broker.order_created observed within 10m of the action
  • expected effect
    Could not be checked: broker was not reachable
    broker.order_filled observed within 10m of the action

Run it in the Failure Lab

The Failure Lab replays each run above: the agent's calls, the records the systems held and the verdict. No account is needed. The verdict it shows is the one recorded with the run: the run turns on a system that could not be read, and the dry run the lab sends cannot say a system was down.

The sandbox in the app runs the exact scenario in your own workspace, as a verified action: