Evaluation checklist
Thirty questions that separate a demo from a product.
Use them on any platform that says it predicts, decides or explains, us included. They are the questions we would want a buyer to ask. A vendor that cannot answer one has told you something.
The model
Before you trust a forecast
- How far ahead is it right, and who chose that distance: you or the vendor?
- What is the baseline it is compared against? Is it “assume nothing changes”, or the vendor’s previous release?
- Show the error at each distance ahead, not an average. Which distances ahead does it lose at?
- Which horizons were never measured, and are they reported as not measured rather than as passes?
- What happens when the model is wrong at the time frame you need? What tells you?
The decision
Before you act on it
- Does every recommendation carry its cost and the cost of waiting?
- Is “do nothing” a priced option, or the default?
- Can the platform tell you when an action costs more than it saves?
- Is the ordering by what each case costs if ignored, or by how unusual it looks?
- What does a recommendation cost you in approval time today, and what would it take to shorten it?
The evidence
Before you defend it
- Is any of the answer generated text? If so, what stops it inventing something?
- Will the same inputs give the same conclusion tomorrow?
- Show the chain from a conclusion back to the events behind it.
- When two sources disagree about the same item, what does the platform show?
- How much of the answer came from the learned component, and who controls that weighting?
Governance and security
Before your reviewer asks
- Where is tenant isolation enforced: in the application, or below it?
- What stops an unvalidated model reaching production? Show a refusal and its recorded reason, and who may override it.
- How do you detect a model trained on its own output?
- What does the platform need write access to? Is there any background write path?
- Which certifications are held today, with dates, and which are only in progress?
Deployment and latency
Before you plan the rollout
- Can it run in your own cloud account or network, and what are the supported topologies?
- Where does your state live, and who else can reach it?
- How do events arrive, and what happens to the ones that fail to map?
- What is measured, and under what conditions? Is a latency figure an inference step or a whole decision?
- What is the plan for irregular and missing data: fall back, or extrapolate?
Commercial
Before you sign
- What is the smallest thing you can buy, and what does it prove?
- What does it cost to be wrong: can you stop?
- Who owns the models trained on your data, and the derived artifacts?
- Is there any path by which your data reaches another customer’s models?
- What is not claimed? Ask for the list, in writing.
We will answer all thirty on your data
Bring the checklist to a working session. The model record, the audit trail and the tenancy answers are all demonstrable.