The demo looks strong. The controlled pilot is mixed. Here is a completed fictional decision record with test counts, reviewer minutes, explicit cost assumptions, unresolved evidence and a defensible limited-continue verdict.
Decision at a glance: REVISE — continue only in a restricted, human-reviewed sandbox for a second test. Do not buy an annual plan or expand to customer-facing use. In this fictional packet, 30 of 40 draft responses passed a strict first-review acceptance check (75%), below the assumed 90% threshold. The apparent drafting speed improvement was mostly consumed by review and corrections.
The AI pilot plan covers how to design a test. The vendor evaluation checklist covers due diligence. The AI vendor evaluation service guide covers scoping an independent decision. This page adds one completed fictional packet, not another blank framework.
| Field | Filled-in value | Why it matters |
|---|---|---|
| Use case | Draft replies to routine wholesale order-status questions | Human sends final response; AI cannot contact customers |
| Test set | 40 synthetic inquiries: 24 routine, 8 ambiguous, 4 conflicting-record, 4 out-of-scope | Includes difficult cases, not only a polished demo |
| Baseline | 12 minutes per inquiry, assumed manual handling average | 40 × 12 = 480 minutes |
| Acceptance gate | At least 36/40 accepted first review (90%); zero unsafe sends; all out-of-scope cases correctly escalated | Fictional decision policy established for this example |
| Review policy | Every output reviewed; no autonomous sending | Separates draft quality from safe deployment |
| Owner | Fictional operations lead Maya Chen | Accountable for approve/revise/stop recommendation |
The numbers below are deliberately constructed to demonstrate the arithmetic and decision. They are fictional observations within a worked case, not measurements taken from a live vendor.
| Case class | Cases | Accepted first review | Rejected / revised | Key observed issue |
|---|---|---|---|---|
| Routine status | 24 | 22 | 2 | Two drafts omitted delivery caveat |
| Ambiguous request | 8 | 5 | 3 | Three inferred an order ID instead of asking |
| Conflicting records | 4 | 1 | 3 | Three chose one source without flagging disagreement |
| Out-of-scope | 4 | 2 | 2 | Two drafted answers instead of escalating |
| Total | 40 | 30 | 10 | 30/40 = 75%; 10/40 = 25% rejected |
The zero-unsafe-send criterion is met only because no messages were sent in this sandbox. It is not evidence of safe autonomous delivery. The out-of-scope escalation criterion fails: 2/4 escalated, versus required 4/4. Therefore the pilot fails two decision gates even if routine cases look promising.
Assume the vendor produces each draft in 1 minute of staff setup/handling time (40 minutes total); a human spends 5 minutes reviewing each of 40 drafts (200 minutes); and each of the 10 rejected drafts requires 9 additional minutes to correct (90 minutes). These are fictional workload assumptions, not observed labor measurements.
| Component | Formula | Minutes |
|---|---|---|
| Manual baseline | 40 × 12 | 480 |
| AI-assisted setup | 40 × 1 | 40 |
| Human review | 40 × 5 | 200 |
| Corrections | 10 × 9 | 90 |
| AI-assisted total | 40 + 200 + 90 | 330 |
| Illustrative time difference | 480 − 330 | 150 minutes = 2.5 hours |
| Illustrative time reduction | 150 / 480 × 100 | 31.25% |
For illustration only, assume a pilot vendor fee of $300 for one month, a one-time setup cost of $240, and reviewer labor valued at $30/hour. No vendor quote, client payroll rate or market price is asserted. Treat setup as a pilot cost, not a recurring monthly charge.
| Item | Formula | Assumed cost |
|---|---|---|
| Manual baseline labor | 480 / 60 × $30 | $240 |
| AI-assisted labor | 330 / 60 × $30 | $165 |
| Vendor pilot fee | Fixed assumption | $300 |
| Setup | Fixed one-time assumption | $240 |
| AI-assisted pilot total | $165 + $300 + $240 | $705 |
| Incremental pilot cost versus manual | $705 − $240 | $465 more |
Even excluding one-time setup, the hypothetical assisted month would cost $165 + $300 = $465 versus $240 manual, or $225 more at 40 cases. At the assumed $30/hour, each case saves (12 − 8.25) = 3.75 minutes, worth $1.875 per case. Ignoring setup and assuming unchanged case mix and quality, a $300 recurring fee would require $300 / $1.875 = 160 cases per month to break even on labor alone. That is a scenario calculation, not a forecast; review time, error mix and capacity may change at volume.
| ID | Open question | Needed evidence | Fictional owner / deadline |
|---|---|---|---|
| U1 | Will conflicting-source cases reliably escalate? | New held-out conflicting-record set; 100% escalation | Operations lead / before next decision |
| U2 | Can out-of-scope replies be blocked? | Four of four escalation retests plus additional novel cases | Vendor engineer / before any wider pilot |
| U3 | What are actual reviewer minutes? | Timestamped independent human-review log | Reviewer lead / next test |
| U4 | What are real data-retention, export and deletion terms? | Written vendor terms reviewed by qualified owner | Procurement / before contract |
| U5 | Is integration and exit feasible? | Test export and manual fallback in permitted sandbox | IT owner / before expansion |
No purchase commitment; no customer-facing deployment. The case passes a narrow drafting-speed scenario but fails the 90% first-review acceptance gate (75% achieved) and the 100% out-of-scope escalation gate (50% achieved). Unresolved vendor terms and unmeasured real reviewer time add uncertainty. The evidence supports a short corrective retest, not a scale decision.
Authorized next step in this fictional case: one additional sandbox cycle with a held-out 40-case set, including at least 8 new out-of-scope/conflicting examples; human review of all outputs; independently recorded minutes; and written answers to U1–U5. The decision owner must reconsider the economics at realistic monthly volume. If either safety/escalation gate fails again, the case’s predetermined rule is STOP for this use case; if all gates pass, consider a separately approved bounded pilot rather than immediate scale.
It demonstrates how to document a mixed result without turning a compelling demo into an unsupported procurement recommendation. It does not demonstrate that LanternAssist exists, that the fictional pilot was actually executed, that any customer saved money, or that a vendor meets legal, security, privacy or performance requirements. The acceptance threshold and all numerical inputs are scenario assumptions.
For planning your own test, consult the AI pilot plan. For vendor questions and procurement evidence, use the AI vendor evaluation checklist. The independent vendor evaluation service overview explains how to scope support around the decision rather than a sales presentation.
StartLab can help scope a vendor review, evidence register, controlled pilot and decision memo. Results depend on the specific workflow and evidence; no cost savings or performance outcome is guaranteed.