AI & automation · Executed synthetic drill

AI Automation Failure Drill: Detect, Contain, Recover, Verify

A human handoff disappears. Would anyone know? This isolated exercise injects the failure, records the observable signals, tests containment, replays safely, and repeats the check.

The proof is the mismatch between the expected human task and the actual task count—and the observable alert before any false completion. A written escalation policy is not evidence that an escalation will fire. This article records one isolated run, a deliberate failure, a recovery, and a retest. For workflow design and SOP definitions, use the existing SOP automation technical checklist and marketing automation SOP template; this drill does not repeat them.

1. Test boundary and pass criteria

The synthetic input is a service inquiry requiring a human to review it. The simulated handoff adapter normally creates exactly one task keyed by the inquiry ID. The injected fault deliberately drops the task-creation action. A separate checker inspects the task store, raises a missing-acknowledgment alert, and blocks the inquiry from being marked complete. Recovery replays the same key; retest uses both a healthy input and another dropped handoff.

Test contract · Defined before observationAll values synthetic
Condition Expected evidence Failure verdict if
Normal handoff One pending human task, state awaiting_human Zero or duplicate tasks
Injected lost handoff Zero tasks detected; missing_ack alert; state blocked Silence or false completed state
Recovery Exactly one task after replay with same key; state awaiting_human Duplicate task or unresolved block
Retest Healthy path succeeds and second injected drop is independently detected Either condition fails

2. Executed inputs and injected failure

In the isolated in-memory run, the harness used a queue dictionary, a task dictionary keyed by synthetic inquiry ID, and an alert list. The submission function writes an awaiting_human state. When drop=True, it skips task creation; the checker then compares expected and actual task existence, appends missing_ack:ID, and changes state to blocked. Recovery uses setdefault on the same ID to avoid duplicate tasks. No external network, human notification, or customer data was involved.

Event log · Actual observed simulation outputLogical order, not wall-clock SLA
Step Input / injected action Expected Actual observed Verdict
E01 Submit SYN-104 with drop=True Task creation fails; checker catches missing task Queue=blocked; tasks=0; alerts=[missing_ack:SYN-104] PASS: failure detected
E02 Recovery: replay SYN-104 with same key Exactly one task; no false completion Queue=awaiting_human; tasks=1 PASS: recovery
E03 Retest SYN-105 with normal route Human task exists; awaiting_human Queue=awaiting_human; task exists PASS: normal route
E04 Retest SYN-106 with drop=True Independent alert and blocked state Queue=blocked; total tasks=2 (SYN-104 and SYN-105); alerts=[missing_ack:SYN-104, missing_ack:SYN-106] PASS: repeated detection
Important scope limit: the simulated checker runs synchronously after each submission. This proves the toy checker noticed a missing task in this executed run. It does not prove a real independent watchdog, external alert delivery, human acknowledgment, or time-bound detection in production. Those require separate controlled tests.

3. Detect, contain, recover, verify: expected versus actual

Detect · PASS in simulationExpected a missing-task signal. Actual: missing_ack:SYN-104 recorded with zero tasks.
Contain · PASS in simulationExpected no silent completion. Actual: inquiry SYN-104 was blocked, not completed.
Recover · PASS in simulationExpected one pending task after replay. Actual: one task existed and the inquiry returned to awaiting_human.
Verify and retest · PASS in simulationExpected healthy routing plus a repeat fault detection. Actual: SYN-105 routed; SYN-106 blocked and alerted.

4. Overall verdict and what was not proved

Simulation verdict: PASS for the four defined checks

One injected handoff loss was noticed, containment prevented a false completion, replay created one task, and a second injected loss was noticed on retest. This is evidence about the isolated program that was run—not evidence that any StartLab or client production automation has passed a failure drill.

There was no actual human acceptance, email/SMS delivery, cross-process watchdog, database durability, restart, queue race, or end-to-end timing measurement. In a real deployment, a useful next drill would separate the watcher from the handoff worker, inject a network timeout or queue outage, and retain timestamped monitoring and human receipt evidence. Do not call an alert delivered merely because an alert object was appended to memory.

5. Reproduce the exercise without touching client systems

Use a local Python interpreter with no third-party packages or credentials. The following is the same essential test logic used in the isolated run. It intentionally uses fictional IDs and in-memory structures only.

Minimal reproduction · Local PythonCopy into a local scratch file
queue = {}; tasks = {}; alerts = []
def submit(k, drop=False):
    queue[k] = 'awaiting_human'
    if not drop:
        tasks.setdefault(k, 'pending')
    if k not in tasks:
        alerts.append('missing_ack:' + k)
        queue[k] = 'blocked'
def recover(k):
    tasks.setdefault(k, 'pending')
    queue[k] = 'awaiting_human'
submit('SYN-104', True)
assert (queue['SYN-104'], len(tasks), alerts[:]) == ('blocked', 0, ['missing_ack:SYN-104'])
recover('SYN-104')
assert (queue['SYN-104'], len(tasks)) == ('awaiting_human', 1)
submit('SYN-105')
submit('SYN-106', True)
assert queue['SYN-105'] == 'awaiting_human'
assert queue['SYN-106'] == 'blocked'
assert len(tasks) == 2
assert alerts == ['missing_ack:SYN-104', 'missing_ack:SYN-106']
print('PASS: detect, contain, recover, verify, retest')

Changing the checker to do nothing is a negative-control experiment: the dropped handoff would remain undetected and the detection assertion would fail. That negative control is suggested for a future run; it was not executed in the run documented here.

6. Evidence a production owner should request

To extend beyond this toy test, ask for an independently captured event stream showing a request ID, expected task ID, missing acknowledgment threshold, alert delivery receipt, named human acknowledgment, containment state, replay key, resulting task ID, and final verification. Run the drill in a permitted sandbox or test tenant, never by silently injecting faults into a client production system.

For design patterns and responsibilities rather than executed proof, refer back to the workflow specification checklist and the marketing handoff SOP.

Want to test the failure path before depending on automation?

StartLab can discuss a bounded test plan, review handoff evidence, and identify what remains unverified. A Strategic Session is a place to scope the work; no uptime, detection-time, or business-result guarantee is implied.

Share this:

Like this:

Like Loading…