Skip to content
All failure modes
Failure mode

Partial completion

Injected failure: downstream step fails, agent claims success. A downstream system returns 500; the agent reports success anyway.

Mode partial_completion of the failure matrix, on all eight workflows. Each run below is the sandbox's scripted agent acting on the sandbox's fake systems of record under the injected failure, an oracle reading those systems' databases, and the verdict of engine 1.0.0 from the systems of record alone.

Synthetic demo data.

Last changed Markdown versionllms.txt

WorkflowAgent reportedOracleCommitLayerWhy
Refund a customersuccessFAILFAILOrder did not move to refunded
Cancel a subscriptionsuccessFAILFAILBilling was not stopped
Change a shipping addresssuccessFAILFAILNo CRM note was added
Approve a discountsuccessFAILFAILNo CRM note was added
Close a support casesuccessFAILFAILNo CRM note was added
Send a payoutsuccessFAILFAILLedger entry was not posted
Deploy a servicesuccessFAILFAILNo healthy check after the deploy
Place a tradesuccessFAILFAILNo CRM note was added

Refund a customer

Scenario refund/partial_completion: downstream step fails, agent claims success.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /tickets/tick_1200
  • GET /orders/order_1001200
  • POST /payments/refunds200
  • POST /orders/order_1001/status500injected: error

Agent notes: tool error: {"error":"error","status":500}.

What the systems of record showed

Records written after the action:

  • payments.refund_created on order_1001
    amount=180, currency=USD

The oracle, reading the databases directly: FAIL

missing orders.order_status_changed; missing crm.note_added; missing tickets.customer_notified

The verdict and why

FAIL Order did not move to refunded

Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.

  • expected effect
    Order did not move to refunded
    orders.order_status_changed observed within 10m of the action where status=refunded
  • expected effect
    No CRM note was added
    crm.note_added observed within 10m of the action
  • expected effect
    Customer was not notified
    tickets.customer_notified observed within 10m of the action

Three models on the same injected failure

From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.

  • AnthropicAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
  • OpenAIAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
  • GoogleAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS

Cancel a subscription

Scenario cancel/partial_completion: downstream step fails, agent claims success.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /crm/subscriptions/sub_1200
  • POST /crm/subscriptions/sub_1/cancel200
  • POST /payments/billing/stop500injected: error

Agent notes: tool error: {"error":"error","status":500}.

What the systems of record showed

Records written after the action:

  • crm.subscription_cancelled on sub_1
    plan=pro

The oracle, reading the databases directly: FAIL

missing payments.billing_stopped; missing tickets.customer_notified

The verdict and why

FAIL Billing was not stopped

Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.

  • expected effect
    Billing was not stopped
    payments.billing_stopped observed within 10m of the action
  • expected effect
    Customer was not notified
    tickets.customer_notified observed within 10m of the action

Three models on the same injected failure

From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.

  • AnthropicAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
  • OpenAIAgent reported success · trace-only judge saidFAIL· truth FAIL · CommitLayer FAIL
  • GoogleAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS

Change a shipping address

Scenario address/partial_completion: downstream step fails, agent claims success.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /orders/order_1001200
  • POST /orders/order_1001/address200
  • POST /crm/notes500injected: error

Agent notes: tool error: {"error":"error","status":500}.

What the systems of record showed

Records written after the action:

  • orders.address_updated on order_1001
    address=22 New Street, Lyon

The oracle, reading the databases directly: FAIL

missing crm.note_added; missing tickets.customer_notified

The verdict and why

FAIL No CRM note was added

Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.

  • expected effect
    No CRM note was added
    crm.note_added observed within 10m of the action
  • expected effect
    Customer was not notified
    tickets.customer_notified observed within 10m of the action

Three models on the same injected failure

From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.

  • AnthropicAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
  • OpenAIAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
  • GoogleAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS

Approve a discount

Scenario discount/partial_completion: downstream step fails, agent claims success.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /orders/order_1004200
  • POST /orders/order_1004/discount200
  • POST /crm/notes500injected: error

Agent notes: tool error: {"error":"error","status":500}.

What the systems of record showed

Records written after the action:

  • orders.discount_applied on order_1004
    percent=15

The oracle, reading the databases directly: FAIL

missing crm.note_added

The verdict and why

FAIL No CRM note was added

Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.

  • expected effect
    No CRM note was added
    crm.note_added observed within 10m of the action

Three models on the same injected failure

From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.

  • AnthropicAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
  • OpenAIAgent reported success · trace-only judge saidFAIL· truth PASS · CommitLayer PASS
  • GoogleAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS

Close a support case

Scenario close/partial_completion: downstream step fails, agent claims success.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /tickets/tick_1200
  • POST /tickets/tick_1/close200
  • POST /crm/notes500injected: error

Agent notes: tool error: {"error":"error","status":500}.

What the systems of record showed

Records written after the action:

  • tickets.ticket_closed on tick_1
    order_id=order_1001

The oracle, reading the databases directly: FAIL

missing crm.note_added; missing tickets.customer_notified

The verdict and why

FAIL No CRM note was added

Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.

  • expected effect
    No CRM note was added
    crm.note_added observed within 10m of the action
  • expected effect
    Customer was not notified
    tickets.customer_notified observed within 10m of the action

Three models on the same injected failure

From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.

  • AnthropicAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
  • OpenAIAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
  • GoogleAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS

Send a payout

Scenario payout/partial_completion: downstream step fails, agent claims success.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /ledger/accounts/acc_1200
  • POST /payments/payouts200
  • POST /ledger/accounts/acc_1/post500injected: error

Agent notes: tool error: {"error":"error","status":500}.

What the systems of record showed

Records written after the action:

  • payments.payout_created on acc_1
    amount=900, currency=USD, payout_id=po_acc_1_1

The oracle, reading the databases directly: FAIL

missing ledger.ledger_posted; missing crm.note_added

The verdict and why

FAIL Ledger entry was not posted

Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.

  • expected effect
    Ledger entry was not posted
    ledger.ledger_posted observed within 10m of the action
  • expected effect
    No CRM note was added
    crm.note_added observed within 10m of the action

Deploy a service

Scenario deploy/partial_completion: downstream step fails, agent claims success.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /platform/services/svc_api200
  • POST /platform/services/svc_api/deploy200
  • POST /platform/services/svc_api/healthcheck500injected: error

Agent notes: tool error: {"error":"error","status":500}.

What the systems of record showed

Records written after the action:

  • platform.deploy_created on svc_api
    version=1.5.0, environment=production

The oracle, reading the databases directly: FAIL

missing platform.health_check; missing crm.note_added

The verdict and why

FAIL No healthy check after the deploy

Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.

  • expected effect
    No healthy check after the deploy
    platform.health_check observed within 10m of the action where status=healthy
  • expected effect
    No change note was added
    crm.note_added observed within 10m of the action

Place a trade

Scenario trade/partial_completion: downstream step fails, agent claims success.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /broker/products/TSLA-USD200
  • POST /broker/orders200
  • GET /broker/orders/ord_TSLA-USD_1200
  • POST /crm/notes500injected: error

Agent notes: tool error: {"error":"error","status":500}.

What the systems of record showed

Records written after the action:

  • broker.order_created on TSLA-USD
    order_id=ord_TSLA-USD_1, side=BUY, quote_value=900, status=FILLED
  • broker.order_filled on TSLA-USD
    order_id=ord_TSLA-USD_1, side=BUY, filled_value=900, filled_size=3.4351145, average_filled_price=262

The oracle, reading the databases directly: FAIL

missing crm.note_added

The verdict and why

FAIL No CRM note was added

Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.

  • expected effect
    No CRM note was added
    crm.note_added observed within 10m of the action

Run it in the Failure Lab

The Failure Lab replays each run above: the agent's calls, the records the systems held and the verdict. No account is needed. The verdict comes from the production engine while you watch.

The sandbox in the app runs the exact scenario in your own workspace, as a verified action: