Questions
Which vendor, and what would make you switch later?
Choose the vendor whose evidence best meets criteria you weighted before the first demo, at a cost you have worked out over the whole contract, with an exit you could afford to use. The demo deserves the least weight of all, since it shows the product on the vendor's own data in a room the vendor arranged.
Set the test before the pitches
Split your requirements in two. Must-haves are pass or fail, such as where your data is stored or a system you must connect to, and a vendor that fails one leaves the list whatever else it offers. Everything else becomes a criterion with a weight, and the weights add up to 100 before any vendor presents.
Then give every vendor the same script, ideally using a sample of your own data. Ask each for two references of similar size and use, and ask those references what went wrong in the third month rather than whether they are happy. Price the full term, including setup, overage, support tiers, and how far the price may rise at renewal.
A worked example
A services firm compares three scheduling vendors with weights of 35 for fit to the workflow, 25 for three-year cost, 20 for implementation risk, 10 for support, and 10 for ease of leaving. Vendor A gave the best demo but offered no reference of similar size. Vendor B costs $70,000 a year plus $40,000 to implement, and two references confirmed go-live within four months. Vendor C is cheapest at $55,000 a year, but its renewal price is uncapped and its data export is a set of PDF files.
Scored out of ten on each criterion and weighted, B leads with 69.5 of 100, C follows at 62.5, and A trails at 61.5 because its implementation score rests on a promise. If C accepts a renewal cap and a full export in a standard format, its exit score rises and it closes to within two points. That single negotiation is worth more than another round of demos.
The usual mistakes
The most common is writing the criteria after the demos, which quietly turns them into a description of the vendor everyone liked. The second is comparing first-year prices when the contract runs three years. The third is treating references the vendor chose as independent evidence, when they are its happiest customers by design.
How the room weighs vendors
Each vendor on the shortlist gets its own Champion, and the criteria and weights are fixed before any option is scored. Inspector separates what was shown from what was said: a demo supports a claim only partly, a reference call confirming a go-live date supports it more, and a brochure promise stays unverified. Scout checks the public facts, such as published terms and documentation.
Gray Wolf hunts the fees and clauses the quote leaves out, and Grandfather prices the way back if the choice goes wrong. When a new requirement surfaces after scoring, Challenge the Verdict runs an independent second pass instead of a quiet edit. The Human Decision, and any approval your procurement rules require, stay with your team.
Questions
- How many vendors belong on a shortlist?
- Usually three. Fewer gives you no real comparison, and more spreads your team's attention so thin that each vendor gets a shallow look.
- Should price be a criterion or a limit?
- Both, in different senses. Set the ceiling you cannot exceed as a must-have, then weight the total cost below that ceiling like any other criterion.
- What if the cheapest vendor wins?
- Then check that it won on the evidence rather than on the gaps. A low price paired with unverified claims and a hard exit is often a high price paid later.
Further reading
Bring Argumont a Decision worth challenging.
Start with a paid 60 day pilot on your own real Decisions.