Choosing an AI implementation partner is one of the few technology decisions where the wrong call costs you twice. Once when the first build under-delivers, and again when you rebuild it with someone else. Enterprises that switch AI vendors after an initial deployment typically lose six to twelve months of productivity and spend 40–60% of the original implementation cost re-architecting and redeploying. The price on the contract is rarely where the real money is decided.
We sit on both sides of this — we deliver implementations, and we are frequently brought in to rescue ones that stalled. The pattern in the rescues is consistent: the partner was chosen on the strength of a polished demo and a client logo wall, and nobody scored the things that actually predict delivery. Below is the 15-question scorecard we hand CIOs and heads of data before they sign. Score each question 0–3. Anything under 33 of 45, keep looking.
Group 1 — Discovery quality (do they understand the problem?)
The single most reliable signal of a capable partner is the quality of the questions they ask before they pitch. Vendors who lead with their product are selling. Vendors who interrogate your process are diagnosing. You want the diagnostician.
- Do they ask about your data before they talk about models? A serious partner wants to know where your data lives, who owns it, how clean it is, and what the access controls look like — because that is where 60% of the effort goes.
- Do they ask about exceptions and edge cases? Strong partners ask how your front-line staff handle the weird 5% — the disputes, the seasonal spikes, the cases that do not fit the form. That is where AI systems break.
- Do they define success with you, in numbers, up front? Resolution rate, cost per interaction, cycle time, error rate — if they cannot help you write a baseline metric in the first two meetings, they will not be able to prove value later.
Group 2 — Delivery evidence (have they actually shipped?)
- Can they give you a production reference you can call? Not a logo, not a press release — a named person at a company who runs the system in production and will take your call. Demos prove nothing about production.
- Will the people in the room be the people who deliver? Ask who, by name, does the work. A common failure: senior engineers win the deal and juniors deliver it. Get the delivery team named in the statement of work.
- Can they show you a system that failed and what they did? Honest partners have war stories. A vendor with a 100% success rate is a vendor who has not shipped much, or is not telling you the truth.
The reference-call question filters more vendors than any other. The strong ones offer references before you ask. The weak ones “need to check with the client” and then go quiet.
Group 3 — Technical portability (are you locked in?)
- Can the system swap foundation models without a rewrite? Model prices and quality change every quarter. If your partner hard-wires a single model’s API into your business logic, you inherit their bet. Insist on an abstraction layer.
- Who owns the weights, the prompts, and the pipelines? If the partner fine-tunes a model on your data, you should own the result. Get IP ownership in writing — including prompts, evaluation sets, and any fine-tuned weights.
- Is it portable across clouds? You may not move clouds next year, but the option has commercial value at renewal time. A partner who builds cloud-portable earns your trust; one who locks you to their preferred vendor is optimising their margin.
Group 4 — Commercial honesty (is the price the real price?)
- Will they write a three-year total cost of ownership? Organisations that build a three-year TCO model before selecting a vendor are 2.8x more likely to stay within budget. The build cost is one line. Run rate, evaluation, monitoring, retraining, and support are the rest.
- Are implementation and run costs separated and explicit? Beware the low build quote with an opaque “managed service” that balloons. Make them break out inference cost assumptions, including what happens at 2x and 5x volume.
- What is the exit plan? A confident partner will tell you exactly how you would leave them — what you keep, how the handover works, what it costs. A partner who gets evasive here is building a moat out of your dependency.
Group 5 — Operating fit (who runs it after go-live?)
- Who owns the system the day after launch? Most AI value is lost in the handover. Clarify whether your team operates it, the partner does, or it is shared — and make sure the model matches your internal capacity.
- How do they handle drift and re-evaluation? Models degrade as the world changes. Ask what their monitoring looks like, how often they re-run the evaluation suite, and who is paged when quality drops.
- Do they transfer knowledge or hoard it? The best partners make themselves progressively less necessary — documentation, runbooks, and training are part of delivery. The worst keep you dependent on tribal knowledge only they hold.
How to use the scorecard
Run it as a structured panel, not a sales call. Send the fifteen questions to your shortlist in advance, then score their answers live. The exercise does two things: it surfaces the weak vendors quickly, and it forces your own team to articulate what success looks like — which is the work you needed to do anyway.
One last filter that no scorecard captures: notice how a partner treats the parts of your problem that are inconvenient for them. The ones who tell you a use case is a bad fit, or that you are not ready, or that a cheaper approach would work — those are the ones worth hiring. The cost of the right partner is bounded. The cost of the wrong one compounds.