Skip to content
Failure modes

Every way the matrix breaks an agent, one page each.

The failure matrix runs 112 scenarios across eight workflows: eight clean runs and 104 runs under an injected failure. Those 104 runs fall into 29 failure modes. Each page says what the agent reported, what the systems of record showed, and the verdict CommitLayer returned and why, then ends in the Failure Lab run of the same scenario.

Synthetic demo data.

Generated from the scenario definitions and a run of the matrix with the sandbox's scripted agent, on the sandbox's fake systems of record, with engine 1.0.0. No page is edited by hand; ten of the page titles are the only words written for them. Last recorded change .

On every workflow

Transport and agent failures the chaos layer injects on each workflow of the matrix.

Failure modeWorkflowsVerdictsLast changed
Lost response, retried
timeout_after_success + retry without idempotency
All eight
FAIL×7PASS
Lost response, idempotent retry
timeout_after_success + idempotent retry
All eight
PASS×8
Phantom success
API returns 200 but nothing changes
All eight
FAIL×8
Partial completion
downstream step fails, agent claims success
All eight
FAIL×8
Permission denied
403 on the primary write, agent claims success
All eight
FAIL×8
Rate limited, then fine
429 once, agent retries
All eight
PASS×8
Success claimed without acting
agent reports success without acting
All eight
FAIL×8
Successful call retried twice
agent retries a successful call twice
All eight
FAIL×7PASS
Stale connector read
connector reads stale data
All eight
PASS×8
System unreachable
a required system is down at verification time
All eight
UNKNOWN×8

Specific to a workflow

Policy limits, preconditions, wrong subjects and records that undo an action, on the workflows where they can happen.

Failure modeWorkflowsVerdictsLast changed
Over the limit
amount above the policy limit; percent above the policy limit
Refund a customer, Approve a discount, Send a payout, Place a trade
FAIL×4
Ineligible order
precondition violated
Refund a customer
FAIL
Wrong customer
agent refunds the wrong order
Refund a customer
FAIL
Concurrent update
a human refunded first
Refund a customer
FAIL
Prompt injection
malicious instruction in ticket text
Refund a customer
FAIL
Forbidden side effect
forbidden side effect
Change a shipping address
FAIL
Frozen account
precondition violated
Send a payout
FAIL
Daily total exceeded
per-agent daily total exceeded
Send a payout, Place a trade
FAIL×2
Amount mismatch
paid amount differs from the request; traded amount differs from the request
Send a payout, Place a trade
FAIL×2
Returned after settlement
payout returned after the verdict settled
Send a payout
FAIL
Degraded service
precondition violated
Deploy a service
FAIL
Wrong environment
parameter not allowed
Deploy a service
FAIL
Too many deploys
per-agent hourly total exceeded
Deploy a service
FAIL
Unhealthy after the deploy
forbidden side effect
Deploy a service
FAIL
Rolled back after settlement
deploy rolled back after the verdict settled
Deploy a service
FAIL
Halted product
precondition violated
Place a trade
FAIL
Wrong product
agent trades the wrong product
Place a trade
FAIL
Order not filled
order accepted but never filled, agent claims success
Place a trade
FAIL
Fill busted after settlement
fill busted after the verdict settled
Place a trade
FAIL