Fictional worked decision · W46

AI Vendor Pilot Evidence Packet: A Worked Go / Revise / Stop Decision

The demo looks strong. The controlled pilot is mixed. Here is a completed fictional decision record with test counts, reviewer minutes, explicit cost assumptions, unresolved evidence and a defensible limited-continue verdict.

Decision at a glance: REVISE — continue only in a restricted, human-reviewed sandbox for a second test. Do not buy an annual plan or expand to customer-facing use. In this fictional packet, 30 of 40 draft responses passed a strict first-review acceptance check (75%), below the assumed 90% threshold. The apparent drafting speed improvement was mostly consumed by review and corrections.

The AI pilot plan covers how to design a test. The vendor evaluation checklist covers due diligence. The AI vendor evaluation service guide covers scoping an independent decision. This page adds one completed fictional packet, not another blank framework.

1. Fictional pilot card and decision thresholds

Packet FVP-09 · Fictional dataControlled, no real customers
Field Filled-in value Why it matters
Use case Draft replies to routine wholesale order-status questions Human sends final response; AI cannot contact customers
Test set 40 synthetic inquiries: 24 routine, 8 ambiguous, 4 conflicting-record, 4 out-of-scope Includes difficult cases, not only a polished demo
Baseline 12 minutes per inquiry, assumed manual handling average 40 × 12 = 480 minutes
Acceptance gate At least 36/40 accepted first review (90%); zero unsafe sends; all out-of-scope cases correctly escalated Fictional decision policy established for this example
Review policy Every output reviewed; no autonomous sending Separates draft quality from safe deployment
Owner Fictional operations lead Maya Chen Accountable for approve/revise/stop recommendation

2. Test register: expected versus recorded result

The numbers below are deliberately constructed to demonstrate the arithmetic and decision. They are fictional observations within a worked case, not measurements taken from a live vendor.

Completed test register · 40 fictional recordsFirst-review acceptance
Case class Cases Accepted first review Rejected / revised Key observed issue
Routine status 24 22 2 Two drafts omitted delivery caveat
Ambiguous request 8 5 3 Three inferred an order ID instead of asking
Conflicting records 4 1 3 Three chose one source without flagging disagreement
Out-of-scope 4 2 2 Two drafted answers instead of escalating
Total 40 30 10 30/40 = 75%; 10/40 = 25% rejected

The zero-unsafe-send criterion is met only because no messages were sent in this sandbox. It is not evidence of safe autonomous delivery. The out-of-scope escalation criterion fails: 2/4 escalated, versus required 4/4. Therefore the pilot fails two decision gates even if routine cases look promising.

3. Reviewer workload: the demo omitted the hard part

Assume the vendor produces each draft in 1 minute of staff setup/handling time (40 minutes total); a human spends 5 minutes reviewing each of 40 drafts (200 minutes); and each of the 10 rejected drafts requires 9 additional minutes to correct (90 minutes). These are fictional workload assumptions, not observed labor measurements.

Workload ledger · Explicit arithmeticAll times assumed
Component Formula Minutes
Manual baseline 40 × 12 480
AI-assisted setup 40 × 1 40
Human review 40 × 5 200
Corrections 10 × 9 90
AI-assisted total 40 + 200 + 90 330
Illustrative time difference 480 − 330 150 minutes = 2.5 hours
Illustrative time reduction 150 / 480 × 100 31.25%
Interpretation: The assumed 31.25% handling-time reduction is not a verified saving, and the 75% acceptance rate plus failed escalation gate block a rollout. A fast draft is not a completed, trustworthy customer response.

4. Cost assumptions and break-even check

For illustration only, assume a pilot vendor fee of $300 for one month, a one-time setup cost of $240, and reviewer labor valued at $30/hour. No vendor quote, client payroll rate or market price is asserted. Treat setup as a pilot cost, not a recurring monthly charge.

Fictional one-month cost ledgerUSD · No revenue or ROI claims
Item Formula Assumed cost
Manual baseline labor 480 / 60 × $30 $240
AI-assisted labor 330 / 60 × $30 $165
Vendor pilot fee Fixed assumption $300
Setup Fixed one-time assumption $240
AI-assisted pilot total $165 + $300 + $240 $705
Incremental pilot cost versus manual $705 − $240 $465 more

Even excluding one-time setup, the hypothetical assisted month would cost $165 + $300 = $465 versus $240 manual, or $225 more at 40 cases. At the assumed $30/hour, each case saves (12 − 8.25) = 3.75 minutes, worth $1.875 per case. Ignoring setup and assuming unchanged case mix and quality, a $300 recurring fee would require $300 / $1.875 = 160 cases per month to break even on labor alone. That is a scenario calculation, not a forecast; review time, error mix and capacity may change at volume.

5. Unresolved questions and evidence owners

Open-issues register · FictionalNo unsupported answers filled in
ID Open question Needed evidence Fictional owner / deadline
U1 Will conflicting-source cases reliably escalate? New held-out conflicting-record set; 100% escalation Operations lead / before next decision
U2 Can out-of-scope replies be blocked? Four of four escalation retests plus additional novel cases Vendor engineer / before any wider pilot
U3 What are actual reviewer minutes? Timestamped independent human-review log Reviewer lead / next test
U4 What are real data-retention, export and deletion terms? Written vendor terms reviewed by qualified owner Procurement / before contract
U5 Is integration and exit feasible? Test export and manual fallback in permitted sandbox IT owner / before expansion

6. Decision memo: REVISE, not GO

Fictional decision: limited continue in sandbox only

No purchase commitment; no customer-facing deployment. The case passes a narrow drafting-speed scenario but fails the 90% first-review acceptance gate (75% achieved) and the 100% out-of-scope escalation gate (50% achieved). Unresolved vendor terms and unmeasured real reviewer time add uncertainty. The evidence supports a short corrective retest, not a scale decision.

Authorized next step in this fictional case: one additional sandbox cycle with a held-out 40-case set, including at least 8 new out-of-scope/conflicting examples; human review of all outputs; independently recorded minutes; and written answers to U1–U5. The decision owner must reconsider the economics at realistic monthly volume. If either safety/escalation gate fails again, the case’s predetermined rule is STOP for this use case; if all gates pass, consider a separately approved bounded pilot rather than immediate scale.

GO — not supportedQuality and escalation gates failed. No autonomous rollout.
REVISE — selectedOne restricted retest to determine whether correctable controls fix the misses.
STOP — conditional next verdictStop this use case if repeated escalation or acceptance thresholds fail.
Unknown — preserve explicitlyData terms, export, real review times and volume economics remain unresolved.

7. What this packet proves—and does not

It demonstrates how to document a mixed result without turning a compelling demo into an unsupported procurement recommendation. It does not demonstrate that LanternAssist exists, that the fictional pilot was actually executed, that any customer saved money, or that a vendor meets legal, security, privacy or performance requirements. The acceptance threshold and all numerical inputs are scenario assumptions.

For planning your own test, consult the AI pilot plan. For vendor questions and procurement evidence, use the AI vendor evaluation checklist. The independent vendor evaluation service overview explains how to scope support around the decision rather than a sales presentation.

Need a defensible AI vendor decision?

StartLab can help scope a vendor review, evidence register, controlled pilot and decision memo. Results depend on the specific workflow and evidence; no cost savings or performance outcome is guaranteed.

Share this:

Like this:

Like Loading…