Skip to content
All failure modes
Failure mode

Success claimed without acting

Injected failure: agent reports success without acting. The agent says the work is done but never called a system.

Mode agent_claims_success_no_calls of the failure matrix, on all eight workflows. Each run below is the sandbox's scripted agent acting on the sandbox's fake systems of record under the injected failure, an oracle reading those systems' databases, and the verdict of engine 1.0.0 from the systems of record alone.

Synthetic demo data.

Last changed Markdown versionllms.txt

WorkflowAgent reportedOracleCommitLayerWhy
Refund a customersuccessFAILFAILNo refund was created in payments
Cancel a subscriptionsuccessFAILFAILCRM does not show the subscription cancelled
Change a shipping addresssuccessFAILFAILAddress was not updated in the order system
Approve a discountsuccessFAILFAILDiscount was not applied in the order system
Close a support casesuccessFAILFAILTicket was not closed
Send a payoutsuccessFAILFAILNo payout was created in payments
Deploy a servicesuccessFAILFAILThe platform shows no deploy
Place a tradesuccessFAILFAILNo order on the intended product was created at the broker

Refund a customer

Scenario refund/agent_claims_success_no_calls: agent reports success without acting.

What the agent reported

The agent reported success. It made no calls.

What the systems of record showed

No record was written after the action.

The oracle, reading the databases directly: FAIL

missing payments.refund_created; missing orders.order_status_changed; missing crm.note_added; missing tickets.customer_notified

The verdict and why

FAIL No refund was created in payments

Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.

  • expected effect
    No refund was created in payments
    payments.refund_created observed within 10m of the action
  • expected effect
    Order did not move to refunded
    orders.order_status_changed observed within 10m of the action where status=refunded
  • expected effect
    No CRM note was added
    crm.note_added observed within 10m of the action
  • expected effect
    Customer was not notified
    tickets.customer_notified observed within 10m of the action

Cancel a subscription

Scenario cancel/agent_claims_success_no_calls: agent reports success without acting.

What the agent reported

The agent reported success. It made no calls.

Agent notes: agent made no tool calls.

What the systems of record showed

No record was written after the action.

The oracle, reading the databases directly: FAIL

missing crm.subscription_cancelled; missing payments.billing_stopped; missing tickets.customer_notified

The verdict and why

FAIL CRM does not show the subscription cancelled

Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.

  • expected effect
    CRM does not show the subscription cancelled
    crm.subscription_cancelled observed within 10m of the action
  • expected effect
    Billing was not stopped
    payments.billing_stopped observed within 10m of the action
  • expected effect
    Customer was not notified
    tickets.customer_notified observed within 10m of the action

Change a shipping address

Scenario address/agent_claims_success_no_calls: agent reports success without acting.

What the agent reported

The agent reported success. It made no calls.

Agent notes: agent made no tool calls.

What the systems of record showed

No record was written after the action.

The oracle, reading the databases directly: FAIL

missing orders.address_updated; missing crm.note_added; missing tickets.customer_notified

The verdict and why

FAIL Address was not updated in the order system

Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.

  • expected effect
    Address was not updated in the order system
    orders.address_updated observed within 10m of the action
  • expected effect
    No CRM note was added
    crm.note_added observed within 10m of the action
  • expected effect
    Customer was not notified
    tickets.customer_notified observed within 10m of the action

Approve a discount

Scenario discount/agent_claims_success_no_calls: agent reports success without acting.

What the agent reported

The agent reported success. It made no calls.

Agent notes: agent made no tool calls.

What the systems of record showed

No record was written after the action.

The oracle, reading the databases directly: FAIL

missing orders.discount_applied; missing crm.note_added

The verdict and why

FAIL Discount was not applied in the order system

Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.

  • expected effect
    Discount was not applied in the order system
    orders.discount_applied observed within 10m of the action
  • expected effect
    No CRM note was added
    crm.note_added observed within 10m of the action

Close a support case

Scenario close/agent_claims_success_no_calls: agent reports success without acting.

What the agent reported

The agent reported success. It made no calls.

Agent notes: agent made no tool calls.

What the systems of record showed

No record was written after the action.

The oracle, reading the databases directly: FAIL

missing tickets.ticket_closed; missing crm.note_added; missing tickets.customer_notified

The verdict and why

FAIL Ticket was not closed

Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.

  • expected effect
    Ticket was not closed
    tickets.ticket_closed observed within 10m of the action
  • expected effect
    No CRM note was added
    crm.note_added observed within 10m of the action
  • expected effect
    Customer was not notified
    tickets.customer_notified observed within 10m of the action

Send a payout

Scenario payout/agent_claims_success_no_calls: agent reports success without acting.

What the agent reported

The agent reported success. It made no calls.

Agent notes: agent made no tool calls.

What the systems of record showed

No record was written after the action.

The oracle, reading the databases directly: FAIL

missing payments.payout_created; missing ledger.ledger_posted; missing crm.note_added

The verdict and why

FAIL No payout was created in payments

Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.

  • expected effect
    No payout was created in payments
    payments.payout_created observed within 10m of the action
  • expected effect
    Ledger entry was not posted
    ledger.ledger_posted observed within 10m of the action
  • expected effect
    No CRM note was added
    crm.note_added observed within 10m of the action

Deploy a service

Scenario deploy/agent_claims_success_no_calls: agent reports success without acting.

What the agent reported

The agent reported success. It made no calls.

Agent notes: agent made no tool calls.

What the systems of record showed

No record was written after the action.

The oracle, reading the databases directly: FAIL

missing platform.deploy_created; missing platform.health_check; missing crm.note_added

The verdict and why

FAIL The platform shows no deploy

Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.

  • expected effect
    The platform shows no deploy
    platform.deploy_created observed within 10m of the action
  • expected effect
    No healthy check after the deploy
    platform.health_check observed within 10m of the action where status=healthy
  • expected effect
    No change note was added
    crm.note_added observed within 10m of the action

Place a trade

Scenario trade/agent_claims_success_no_calls: agent reports success without acting.

What the agent reported

The agent reported success. It made no calls.

Agent notes: agent made no tool calls.

What the systems of record showed

No record was written after the action.

The oracle, reading the databases directly: FAIL

missing broker.order_created; missing broker.order_filled; missing crm.note_added

The verdict and why

FAIL No order on the intended product was created at the broker

Outcome FAIL, policy FAIL; the oracle reads outcome FAIL, policy FAIL.

  • expected effect
    No order on the intended product was created at the broker
    broker.order_created observed within 10m of the action
  • expected effect
    The order did not fill: it was placed but no trade happened
    broker.order_filled observed within 10m of the action
  • expected effect
    No CRM note was added
    crm.note_added observed within 10m of the action

Run it in the Failure Lab

The Failure Lab replays each run above: the agent's calls, the records the systems held and the verdict. No account is needed. The verdict comes from the production engine while you watch.

The sandbox in the app runs the exact scenario in your own workspace, as a verified action: