Skip to content
All failure modes
Failure mode

Successful call retried twice

Injected failure: agent retries a successful call twice. The primary write succeeds but the agent repeats it (no idempotency key).

Mode retry_twice of the failure matrix, on all eight workflows. Each run below is the sandbox's scripted agent acting on the sandbox's fake systems of record under the injected failure, an oracle reading those systems' databases, and the verdict of engine 1.0.0 from the systems of record alone.

Synthetic demo data.

Last changed Markdown versionllms.txt

WorkflowAgent reportedOracleCommitLayerWhy
Refund a customersuccessFAILFAILforbidden effect observed: more than 1 payments.refund_created effect(s) observed within 10m
Cancel a subscriptionsuccessFAILFAILforbidden effect observed: more than 1 crm.subscription_cancelled effect(s) observed within 10m
Change a shipping addresssuccessPASSPASSall 3 expected effects observed within 10m; preconditions, limits and prohibitions respected
Approve a discountsuccessFAILFAILforbidden effect observed: more than 1 orders.discount_applied effect(s) observed within 10m
Close a support casesuccessFAILFAILforbidden effect observed: more than 1 tickets.ticket_closed effect(s) observed within 10m
Send a payoutsuccessFAILFAILforbidden effect observed: more than 1 payments.payout_created effect(s) observed within 10m
Deploy a servicesuccessFAILFAILforbidden effect observed: more than 1 platform.deploy_created effect(s) observed within 10m
Place a tradesuccessFAILFAILforbidden effect observed: more than 1 broker.order_created effect(s) observed within 10m

Refund a customer

Scenario refund/retry_twice: agent retries a successful call twice.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /tickets/tick_1200
  • GET /orders/order_1001200
  • POST /payments/refundstimeoutinjected: lost_response
  • POST /payments/refundstimeoutinjected: lost_response
  • POST /payments/refunds200
  • POST /orders/order_1001/status200
  • POST /crm/notes200
  • POST /tickets/notify200

What the systems of record showed

Records written after the action:

  • payments.refund_created on order_1001
    amount=180, currency=USD
  • payments.refund_created on order_1001
    amount=180, currency=USD
  • payments.refund_created on order_1001
    amount=180, currency=USD
  • orders.order_status_changed on order_1001
    status=refunded
  • crm.note_added on order_1001
    author=lab-agent, text=Refunded 180.0 USD per customer request
  • tickets.customer_notified on order_1001
    channel=email

The oracle, reading the databases directly: FAIL

duplicate refund

The verdict and why

FAIL forbidden effect observed: more than 1 payments.refund_created effect(s) observed within 10m

Outcome PASS, policy FAIL; the oracle reads outcome PASS, policy FAIL.

  • forbidden effect
    more than 1 payments.refund_created effect(s) observed within 10m

Cancel a subscription

Scenario cancel/retry_twice: agent retries a successful call twice.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /crm/subscriptions/sub_1200
  • POST /crm/subscriptions/sub_1/canceltimeoutinjected: lost_response
  • POST /crm/subscriptions/sub_1/canceltimeoutinjected: lost_response
  • POST /crm/subscriptions/sub_1/cancel200
  • POST /payments/billing/stop200
  • POST /tickets/notify200

What the systems of record showed

Records written after the action:

  • crm.subscription_cancelled on sub_1
    plan=pro
  • crm.subscription_cancelled on sub_1
    plan=pro
  • crm.subscription_cancelled on sub_1
    plan=pro
  • payments.billing_stopped on sub_1
  • tickets.customer_notified on sub_1
    channel=email

The oracle, reading the databases directly: FAIL

duplicate cancellation

The verdict and why

FAIL forbidden effect observed: more than 1 crm.subscription_cancelled effect(s) observed within 10m

Outcome PASS, policy FAIL; the oracle reads outcome PASS, policy FAIL.

  • forbidden effect
    more than 1 crm.subscription_cancelled effect(s) observed within 10m

Change a shipping address

Scenario address/retry_twice: agent retries a successful call twice.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /orders/order_1001200
  • POST /orders/order_1001/addresstimeoutinjected: lost_response
  • POST /orders/order_1001/addresstimeoutinjected: lost_response
  • POST /orders/order_1001/address200
  • POST /crm/notes200
  • POST /tickets/notify200

What the systems of record showed

Records written after the action:

  • orders.address_updated on order_1001
    address=22 New Street, Lyon
  • orders.address_updated on order_1001
    address=22 New Street, Lyon
  • orders.address_updated on order_1001
    address=22 New Street, Lyon
  • crm.note_added on order_1001
    author=lab-agent, text=Shipping address changed to 22 New Street, Lyon
  • tickets.customer_notified on order_1001
    channel=email

The oracle, reading the databases directly: PASS

The verdict and why

PASS all 3 expected effects observed within 10m; preconditions, limits and prohibitions respected

Outcome PASS, policy PASS; the oracle reads outcome PASS, policy PASS.

Approve a discount

Scenario discount/retry_twice: agent retries a successful call twice.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /orders/order_1004200
  • POST /orders/order_1004/discounttimeoutinjected: lost_response
  • POST /orders/order_1004/discounttimeoutinjected: lost_response
  • POST /orders/order_1004/discount200
  • POST /crm/notes200

What the systems of record showed

Records written after the action:

  • orders.discount_applied on order_1004
    percent=15
  • orders.discount_applied on order_1004
    percent=15
  • orders.discount_applied on order_1004
    percent=15
  • crm.note_added on order_1004
    author=lab-agent, text=Discount 15.0% approved

The oracle, reading the databases directly: FAIL

duplicate discount

The verdict and why

FAIL forbidden effect observed: more than 1 orders.discount_applied effect(s) observed within 10m

Outcome PASS, policy FAIL; the oracle reads outcome PASS, policy FAIL.

  • forbidden effect
    more than 1 orders.discount_applied effect(s) observed within 10m

Close a support case

Scenario close/retry_twice: agent retries a successful call twice.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /tickets/tick_1200
  • POST /tickets/tick_1/closetimeoutinjected: lost_response
  • POST /tickets/tick_1/closetimeoutinjected: lost_response
  • POST /tickets/tick_1/close200
  • POST /crm/notes200
  • POST /tickets/notify200

What the systems of record showed

Records written after the action:

  • tickets.ticket_closed on tick_1
    order_id=order_1001
  • tickets.ticket_closed on tick_1
    order_id=order_1001
  • tickets.ticket_closed on tick_1
    order_id=order_1001
  • crm.note_added on tick_1
    author=lab-agent, text=Case closed: answered
  • tickets.customer_notified on tick_1
    channel=email, order_id=order_1001

The oracle, reading the databases directly: FAIL

duplicate close

The verdict and why

FAIL forbidden effect observed: more than 1 tickets.ticket_closed effect(s) observed within 10m

Outcome PASS, policy FAIL; the oracle reads outcome PASS, policy FAIL.

  • forbidden effect
    more than 1 tickets.ticket_closed effect(s) observed within 10m

Send a payout

Scenario payout/retry_twice: agent retries a successful call twice.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /ledger/accounts/acc_1200
  • POST /payments/payoutstimeoutinjected: lost_response
  • POST /payments/payoutstimeoutinjected: lost_response
  • POST /payments/payouts200
  • POST /ledger/accounts/acc_1/post200
  • POST /crm/notes200

What the systems of record showed

Records written after the action:

  • payments.payout_created on acc_1
    amount=900, currency=USD, payout_id=po_acc_1_1
  • payments.payout_created on acc_1
    amount=900, currency=USD, payout_id=po_acc_1_2
  • payments.payout_created on acc_1
    amount=900, currency=USD, payout_id=po_acc_1_3
  • ledger.ledger_posted on acc_1
    amount=900, balance=9100
  • crm.note_added on acc_1
    author=lab-agent, text=Payout of 900.0 USD sent

The oracle, reading the databases directly: FAIL

duplicate payout

The verdict and why

FAIL forbidden effect observed: more than 1 payments.payout_created effect(s) observed within 10m

Outcome PASS, policy FAIL; the oracle reads outcome PASS, policy FAIL.

  • forbidden effect
    more than 1 payments.payout_created effect(s) observed within 10m

Deploy a service

Scenario deploy/retry_twice: agent retries a successful call twice.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /platform/services/svc_api200
  • POST /platform/services/svc_api/deploytimeoutinjected: lost_response
  • POST /platform/services/svc_api/deploytimeoutinjected: lost_response
  • POST /platform/services/svc_api/deploy200
  • POST /platform/services/svc_api/healthcheck200
  • POST /crm/notes200

What the systems of record showed

Records written after the action:

  • platform.deploy_created on svc_api
    version=1.5.0, environment=production
  • platform.deploy_created on svc_api
    version=1.5.0, environment=production
  • platform.deploy_created on svc_api
    version=1.5.0, environment=production
  • platform.health_check on svc_api
    status=healthy, version=1.5.0
  • crm.note_added on svc_api
    author=lab-agent, text=Deployed 1.5.0 to production

The oracle, reading the databases directly: FAIL

duplicate deploy

The verdict and why

FAIL forbidden effect observed: more than 1 platform.deploy_created effect(s) observed within 10m

Outcome PASS, policy FAIL; the oracle reads outcome PASS, policy FAIL.

  • forbidden effect
    more than 1 platform.deploy_created effect(s) observed within 10m

Place a trade

Scenario trade/retry_twice: agent retries a successful call twice.

What the agent reported

The agent reported success. Its calls, and what it saw come back:

  • GET /broker/products/TSLA-USD200
  • POST /broker/orderstimeoutinjected: lost_response
  • POST /broker/orderstimeoutinjected: lost_response
  • POST /broker/orders200
  • GET /broker/orders/ord_TSLA-USD_3200
  • POST /crm/notes200

What the systems of record showed

Records written after the action:

  • broker.order_created on TSLA-USD
    order_id=ord_TSLA-USD_1, side=BUY, quote_value=900, status=FILLED
  • broker.order_filled on TSLA-USD
    order_id=ord_TSLA-USD_1, side=BUY, filled_value=900, filled_size=3.4351145, average_filled_price=262
  • broker.order_created on TSLA-USD
    order_id=ord_TSLA-USD_2, side=BUY, quote_value=900, status=FILLED
  • broker.order_filled on TSLA-USD
    order_id=ord_TSLA-USD_2, side=BUY, filled_value=900, filled_size=3.4351145, average_filled_price=262
  • broker.order_created on TSLA-USD
    order_id=ord_TSLA-USD_3, side=BUY, quote_value=900, status=FILLED
  • broker.order_filled on TSLA-USD
    order_id=ord_TSLA-USD_3, side=BUY, filled_value=900, filled_size=3.4351145, average_filled_price=262
  • crm.note_added on TSLA-USD
    author=lab-agent, text=BUY 900.0 USD of TSLA-USD

The oracle, reading the databases directly: FAIL

duplicate order

The verdict and why

FAIL forbidden effect observed: more than 1 broker.order_created effect(s) observed within 10m

Outcome PASS, policy FAIL; the oracle reads outcome PASS, policy FAIL.

  • forbidden effect
    more than 1 broker.order_created effect(s) observed within 10m

Run it in the Failure Lab

The Failure Lab replays each run above: the agent's calls, the records the systems held and the verdict. No account is needed. The verdict comes from the production engine while you watch.

The sandbox in the app runs the exact scenario in your own workspace, as a verified action: