An AI demo can look impressive and still be a poor business purchase. The real question is not whether a vendor can produce a convincing output in a controlled demonstration. It is whether the product can solve a defined business problem with acceptable risk, predictable operating effort, clear data rules, useful evidence, and a practical way to stop using it.
For a small business, vendor evaluation should be rigorous without becoming enterprise procurement theater. You need enough due diligence to expose the decisions that matter: what the system touches, what can go wrong, who reviews the output, how you measure value, what the vendor may do with your information, and what happens if pricing, product behavior, or your needs change.
Before you score the vendor
- Name one specific workflow or use case.
- Define the current baseline: time, cost, quality, errors, or capacity.
- Identify the data the tool would receive.
- Decide which actions must remain human-controlled.
- Set a pilot boundary and a stop condition.
Critical-fail rule
A high total score should not override a serious unresolved issue. Treat unclear data-use terms, unacceptable security gaps, missing deletion/exit options, unsupported claims, or an inability to test your real use case as a hold until resolved.
The 100-point scorecard below is a StartLab decision aid, not a regulatory standard.
The 100-Point AI Vendor Evaluation Scorecard
Confirm the vendor solves the workflow you actually need, not a generic “AI transformation” problem. Evaluate who will use it, what task it replaces or assists, what outcome improves, and where the product should not be used.
Map what business, employee, customer, or confidential information enters the product. Review data-use terms, access controls, retention, deletion, subprocessors, training or model-improvement use, and security responsibilities before uploading sensitive information.
Require evidence that matches your task. Generic benchmark claims do not tell you how the product performs on your customer questions, documents, exception cases, or operational constraints. Test representative and failure cases.
Assess integration, permissions, monitoring, support, owner responsibilities, change management, and the work required after launch. A tool that saves 10 hours but creates 12 hours of review and exception handling is not operational leverage.
Calculate the operating cost, not just the subscription. Include implementation, usage, integrations, review labor, support, overages, migration, and the cost of switching. Confirm data export, deletion, contract termination, and continuity options.
Questions to Ask an AI Vendor Before You Buy
Business and product
- Which use cases are supported in production today?
- Which use cases are still experimental or roadmap items?
- What customer prerequisites make implementation succeed?
- What common reasons cause deployments to fail or stall?
- Which features depend on third-party models or services?
Data and privacy
- What data do you collect from our users and systems?
- Is our data used for training, fine-tuning, product improvement, or evaluation?
- Can we opt out of secondary uses?
- What are the retention and deletion rules?
- Where is data processed and which subprocessors receive it?
Security and access
- What access controls and authentication options are available?
- Can we limit permissions by role or workflow?
- How are data in transit and stored data protected?
- How are security incidents communicated?
- Can you provide current evidence for relevant security controls or independent assessments?
Evaluation and control
- How should we test accuracy or task performance for our use case?
- What known failure modes should we include in testing?
- Can a human review, reject, revise, or override outputs?
- Can we keep logs or evidence needed for troubleshooting?
- How are material model or product changes communicated?
Implementation and support
- What systems and APIs can the product integrate with?
- Who owns setup, workflow design, prompt/configuration changes, and monitoring?
- What support is included versus paid separately?
- What happens when a dependency is unavailable?
- What is the rollback or manual fallback path?
Commercial and exit
- What drives monthly and variable usage cost?
- Are there minimums, overages, seat charges, or implementation fees?
- Can we export our data and configuration in usable formats?
- What is deleted when the contract ends, and when?
- What dependencies make switching difficult?
Why Data Terms Deserve Their Own Vendor Review
The Federal Trade Commission has specifically warned AI model-as-a-service companies that customer data can include confidential internal documents and other sensitive information, and that companies must honor privacy and confidentiality commitments. For buyers, that makes vendor promises about training, retention, sharing, and secondary use a procurement issue—not fine print to read after rollout.
The FTC’s small-business cybersecurity guidance also recommends putting vendor security requirements in writing, specifying how data may be used, shared, sold, retained, and deleted, and establishing a process to verify compliance rather than simply accepting assurances.
That does not mean every small business needs a lengthy security questionnaire. It means the depth of review should match the sensitivity of the data and the consequence of a failure. A tool that only reformats public marketing copy does not create the same exposure as a system connected to customer records, employee data, contracts, financial information, or internal knowledge.
Evaluate the System, Not Just the Demo
NIST’s voluntary AI Risk Management Framework organizes AI risk work around the functions Govern, Map, Measure, and Manage. For vendor selection, the practical lesson is straightforward: understand the context and ownership of the use case, measure system behavior with relevant evidence, and define how risks will be managed before scaling.
NIST’s Generative AI Profile extends that framework for generative AI risks. In August 2026, NIST also released the initial public draft of its TEVV-Athlon framework, a structured approach for testing, evaluation, verification, and validation of AI systems and their real-world outcomes. Small businesses do not need to reproduce a formal laboratory evaluation, but they should borrow the core discipline: test the product against the task you will actually run.
| Weak evaluation | Stronger evaluation |
|---|---|
| “The demo looked accurate.” | Run representative examples, edge cases, and failure cases from your workflow. |
| “The vendor says it is secure.” | Match requested evidence and contract controls to your actual data exposure. |
| “It saves time.” | Compare the baseline with implementation, review, exception, and maintenance effort included. |
| “It integrates with our CRM.” | Test permissions, field mapping, failures, retries, duplicate prevention, and rollback. |
| “It has human review.” | Define who reviews what, when escalation happens, and what actions remain prohibited. |
AI Vendor Red Flags That Should Trigger a Hold
Pause the purchase when a material answer remains unclear
- The vendor cannot explain whether customer inputs are used for training or product improvement.
- Data retention or deletion terms are vague or inconsistent across sales material and contract terms.
- The product needs broader access than the use case requires.
- Performance claims rely on generic benchmarks but the vendor resists testing your real workflow.
- There is no credible fallback when the system, integration, or upstream model fails.
- Human review exists in theory but cannot be placed before a consequential action.
- Pricing becomes difficult to estimate at your expected usage level.
- You cannot export essential data or configurations in a usable form.
- The vendor promises guaranteed business results, savings, or revenue without evidence tied to your baseline and use case.
Recent FTC enforcement also reinforces a basic procurement rule: treat dramatic AI marketing claims as claims to verify, not as evidence. In March 2026, the FTC announced a settlement involving allegations that Air AI misled entrepreneurs and small businesses with earnings and refund claims. The lesson for buyers is broader than one company—require evidence for outcomes that materially affect the purchase decision.
Build a Pilot Packet Before Signing a Long-Term Commitment
Once a vendor clears the evaluation gate, move to a bounded pilot instead of immediate broad deployment. StartLab’s AI pilot plan for small businesses provides the next layer: scope, approved inputs, representative test cases, success metrics, human review, rollback, and a scale-or-stop decision.
Your pilot packet should include:
- One use case. Avoid evaluating multiple unrelated workflows at once.
- A baseline. Record the current time, quality, cost, errors, capacity, or response metric.
- Approved data. Define what the pilot may and may not receive.
- Test cases. Include normal examples, edge cases, and known failure scenarios.
- Review ownership. Name the person responsible for validating outcomes.
- Success and stop criteria. Decide in advance what evidence supports scale, revision, hold, or exit.
- Rollback. Preserve a manual or previous workflow until the pilot earns broader trust.
Use the Lightest Procurement Process That Matches the Risk
A low-risk content assistant may only need a short terms review, bounded data policy, sample testing, and a monthly cost check. A system connected to customer, employee, financial, health, legal, or other sensitive information deserves deeper review and may require qualified legal, security, privacy, or compliance advice appropriate to your industry and jurisdiction.
The goal is not to eliminate all uncertainty. It is to make the uncertainty visible before the tool becomes embedded in operations.
Need Help Deciding What to Evaluate First?
Use StartLab’s free Business Growth Checker to identify whether the real constraint is your website, marketing, sales follow-up, operations, automation, measurement, or AI readiness. If you already have a defined AI initiative, a Strategic Session can help turn the use case into an evaluation and pilot plan.
Frequently Asked Questions
What is the most important question to ask an AI vendor?
Start with the specific business use case and the data required to run it. Without those two facts, you cannot judge product fit, privacy exposure, security needs, evaluation quality, operating effort, or value.
Should a small business ask whether its data is used to train AI models?
Yes when business, customer, employee, or confidential information may enter the service. Review the vendor’s current terms and contract language for training, product improvement, retention, sharing, deletion, and any available controls.
Do certifications prove an AI product is safe for my use case?
No single certification or assessment proves a product is suitable for every workflow. Security evidence can support due diligence, but you still need to evaluate the actual use case, data, permissions, system behavior, human controls, and failure consequences.
How long should an AI vendor pilot run?
Long enough to test a representative sample of normal and failure cases and compare the results with a baseline. The right duration depends on workflow volume and variability; define the evidence you need before choosing a calendar length.
Should I choose the vendor with the highest score?
Not automatically. Use the score to structure comparison, then apply critical-fail gates. An unresolved issue involving data use, security, evaluation, contract terms, or exit may outweigh a high total score.
Authoritative references
- NIST — AI Risk Management Framework
- NIST — Generative AI Profile
- NIST — TEVV-Athlon Framework for Evaluating AI Systems (initial public draft)
- FTC — AI Companies: Uphold Your Privacy and Confidentiality Commitments
- FTC — Cybersecurity for Small Business: Vendor Security
- FTC — March 2026 Air AI settlement announcement
This article is general business guidance, not legal, cybersecurity, privacy, procurement, or compliance advice.