Skip to content
All failure modes
Failure mode

Phantom success

Injected failure: API returns 200 but nothing changes. The primary write answers 200 without writing; the agent's trace looks perfect.

Mode phantom_success of the failure matrix, on all eight workflows. Each run below is the sandbox's scripted agent acting on the sandbox's fake systems of record under the injected failure, an oracle reading those systems' databases, and the verdict of engine 1.0.0 from the systems of record alone.

Synthetic demo data.

Last changed Markdown versionllms.txt

WorkflowAgent reportedOracleCommitLayerWhy
Refund a customersuccessFAILFAILNo refund was created in payments
Cancel a subscriptionsuccessFAILFAILCRM does not show the subscription cancelled
Change a shipping addresssuccessFAILFAILAddress was not updated in the order system
Approve a discountsuccessFAILFAILDiscount was not applied in the order system
Close a support casesuccessFAILFAILTicket was not closed
Send a payoutsuccessFAILFAILNo payout was created in payments
Deploy a servicesuccessFAILFAILThe platform shows no deploy
Place a tradesuccessFAILFAILNo order on the intended product was created at the broker

Refund a customer

Scenario refund/phantom_success: API returns 200 but nothing changes.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /tickets/tick_1200
  • GET /orders/order_1001200
  • POST /payments/refunds200injected: phantom_success
  • POST /orders/order_1001/status200
  • POST /crm/notes200
  • POST /tickets/notify200

What the systems of record showed

Records written after the action:

  • orders.order_status_changed on order_1001
    status=refunded
  • crm.note_added on order_1001
    author=lab-agent, text=Refunded 180.0 USD per customer request
  • tickets.customer_notified on order_1001
    channel=email

The oracle, reading the databases directly: FAIL

missing payments.refund_created

The verdict and why

FAIL No refund was created in payments

Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.

  • expected effect
    No refund was created in payments
    payments.refund_created observed within 10m of the action

Three models on the same injected failure

From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.

  • AnthropicAgent reported failure · trace-only judge saidFAIL· truth FAIL · CommitLayer FAIL
  • OpenAIAgent reported success · trace-only judge saidFAIL· truth FAIL · CommitLayer FAIL
  • GoogleAgent reported success · trace-only judge saidFAIL· truth FAIL · CommitLayer FAIL

Cancel a subscription

Scenario cancel/phantom_success: API returns 200 but nothing changes.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /crm/subscriptions/sub_1200
  • POST /crm/subscriptions/sub_1/cancel200injected: phantom_success
  • POST /payments/billing/stop200
  • POST /tickets/notify200

What the systems of record showed

Records written after the action:

  • payments.billing_stopped on sub_1
  • tickets.customer_notified on sub_1
    channel=email

The oracle, reading the databases directly: FAIL

missing crm.subscription_cancelled

The verdict and why

FAIL CRM does not show the subscription cancelled

Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.

  • expected effect
    CRM does not show the subscription cancelled
    crm.subscription_cancelled observed within 10m of the action

Three models on the same injected failure

From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.

  • AnthropicAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
  • OpenAIAgent reported success · trace-only judge saidFAIL· truth FAIL · CommitLayer FAIL
  • GoogleAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS

Change a shipping address

Scenario address/phantom_success: API returns 200 but nothing changes.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /orders/order_1001200
  • POST /orders/order_1001/address200injected: phantom_success
  • POST /crm/notes200
  • POST /tickets/notify200

What the systems of record showed

Records written after the action:

  • crm.note_added on order_1001
    author=lab-agent, text=Shipping address changed to 22 New Street, Lyon
  • tickets.customer_notified on order_1001
    channel=email

The oracle, reading the databases directly: FAIL

missing orders.address_updated

The verdict and why

FAIL Address was not updated in the order system

Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.

  • expected effect
    Address was not updated in the order system
    orders.address_updated observed within 10m of the action

Three models on the same injected failure

From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.

  • AnthropicAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
  • OpenAIAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
  • GoogleAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS

Approve a discount

Scenario discount/phantom_success: API returns 200 but nothing changes.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /orders/order_1004200
  • POST /orders/order_1004/discount200injected: phantom_success
  • POST /crm/notes200

What the systems of record showed

Records written after the action:

  • crm.note_added on order_1004
    author=lab-agent, text=Discount 15.0% approved

The oracle, reading the databases directly: FAIL

missing orders.discount_applied

The verdict and why

FAIL Discount was not applied in the order system

Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.

  • expected effect
    Discount was not applied in the order system
    orders.discount_applied observed within 10m of the action

Three models on the same injected failure

From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.

  • AnthropicAgent reported failure · trace-only judge saidFAIL· truth FAIL · CommitLayer FAIL
  • OpenAIAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
  • GoogleAgent reported success · trace-only judge saidFAIL· truth FAIL · CommitLayer FAIL

Close a support case

Scenario close/phantom_success: API returns 200 but nothing changes.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /tickets/tick_1200
  • POST /tickets/tick_1/close200injected: phantom_success
  • POST /crm/notes200
  • POST /tickets/notify200

What the systems of record showed

Records written after the action:

  • crm.note_added on tick_1
    author=lab-agent, text=Case closed: answered
  • tickets.customer_notified on tick_1
    channel=email, order_id=order_1001

The oracle, reading the databases directly: FAIL

missing tickets.ticket_closed

The verdict and why

FAIL Ticket was not closed

Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.

  • expected effect
    Ticket was not closed
    tickets.ticket_closed observed within 10m of the action

Three models on the same injected failure

From the benchmark, where a model drove the agent instead of a script. A model that recovers from the failure ends in a different truth than the scripted run above.

  • AnthropicAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
  • OpenAIAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS
  • GoogleAgent reported success · trace-only judge saidPASS· truth PASS · CommitLayer PASS

Send a payout

Scenario payout/phantom_success: API returns 200 but nothing changes.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /ledger/accounts/acc_1200
  • POST /payments/payouts200injected: phantom_success
  • POST /ledger/accounts/acc_1/post200
  • POST /crm/notes200

What the systems of record showed

Records written after the action:

  • ledger.ledger_posted on acc_1
    amount=900, balance=9100
  • crm.note_added on acc_1
    author=lab-agent, text=Payout of 900.0 USD sent

The oracle, reading the databases directly: FAIL

missing payments.payout_created

The verdict and why

FAIL No payout was created in payments

Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.

  • expected effect
    No payout was created in payments
    payments.payout_created observed within 10m of the action

Deploy a service

Scenario deploy/phantom_success: API returns 200 but nothing changes.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /platform/services/svc_api200
  • POST /platform/services/svc_api/deploy200injected: phantom_success
  • POST /platform/services/svc_api/healthcheck200
  • POST /crm/notes200

What the systems of record showed

Records written after the action:

  • platform.health_check on svc_api
    status=healthy, version=1.4.0
  • crm.note_added on svc_api
    author=lab-agent, text=Deployed 1.5.0 to production

The oracle, reading the databases directly: FAIL

missing platform.deploy_created

The verdict and why

FAIL The platform shows no deploy

Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.

  • expected effect
    The platform shows no deploy
    platform.deploy_created observed within 10m of the action

Place a trade

Scenario trade/phantom_success: API returns 200 but nothing changes.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /broker/products/TSLA-USD200
  • POST /broker/orders200injected: phantom_success
  • POST /crm/notes200

What the systems of record showed

Records written after the action:

  • crm.note_added on TSLA-USD
    author=lab-agent, text=BUY 900.0 USD of TSLA-USD

The oracle, reading the databases directly: FAIL

missing broker.order_created; missing broker.order_filled

The verdict and why

FAIL No order on the intended product was created at the broker

Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.

  • expected effect
    No order on the intended product was created at the broker
    broker.order_created observed within 10m of the action
  • expected effect
    The order did not fill: it was placed but no trade happened
    broker.order_filled observed within 10m of the action

Run it in the Failure Lab

The Failure Lab replays each run above: the agent's calls, the records the systems held and the verdict. No account is needed. The verdict comes from the production engine while you watch.

The sandbox in the app runs the exact scenario in your own workspace, as a verified action: