Every analyst publishes the same depressing number. McKinsey, Gartner, IBM, IDC — they all converge somewhere between 65% and 80% of AI pilots that never reach production. The numbers vary by survey methodology, but the direction does not. Most AI work dies in the pilot.
The cynical reading is that AI does not work. We do not buy that. We have shipped too many models that earn more in a quarter than they cost to build in a year. The less cynical reading — supported by our own engagement data and by what we see in the public failure post-mortems — is that pilots die from organisational and delivery failures, not technical ones. Below is the pattern, and the seven practices that materially shift the survival rate.
The pattern
Most pilots die in one of three ways:
- The metric problem. Nobody agreed on what success looks like in production terms. The pilot “works” on the demo data, but no business KPI improved measurably because no business KPI was instrumented.
- The handover problem. The data science team shipped a Jupyter notebook; the engineering team will not run a Jupyter notebook in production. Six months go by trying to make the notebook into a service. Sponsorship evaporates.
- The ownership problem. Once the consultant or vendor leaves, nobody owns the model. It drifts, breaks, embarrasses someone, and gets quietly turned off.
These three problems are not exotic. They are not unique to AI. We have watched them kill ERP rollouts, data lake migrations, and product launches for decades. What is unique to AI is that the demo is so much more impressive than the production reality, which means the gap between expectation and delivery is wider — and the disappointment is sharper.
1. Define the production KPI before you write a line of code
Every engagement should start with a sentence of this form: “In production, this system will succeed if [specific business metric] moves from [baseline] to [target] over [time horizon], measured by [instrumented dashboard].” Without this sentence written down and signed off by an executive sponsor, you do not have a project; you have a science fair.
Model accuracy is not a production KPI. Latency is not a production KPI. They are necessary, but they are inputs to the production KPI — which is almost always a financial or operational number that someone in the business already cares about. Revenue, cost, cycle time, retention, NPS, churn, throughput.
2. Choose use cases where the data is already trustworthy
Half of failed pilots fail because the data is not what the slide deck claimed. Records are missing, fields are inconsistent, definitions disagree across systems. The first 60% of the project becomes data plumbing, by which point the sponsor has lost patience.
Resist the temptation to chase glamorous use cases on shaky data. The first AI project in an organisation should hit a use case where the data is already in use for analytics or reporting. Boring foundations beat ambitious moonshots when you are trying to build AI credibility inside an organisation.
3. Engineer for production from day one
A model that lives in a notebook is not a model — it is a research artefact. Production engineering is a different discipline: containers, registries, observability, drift detection, retraining triggers, rollback strategy, secrets management. Tack this on at the end and you will spend twice as long retrofitting as you would have spent building it in from the start.
Use the engagement’s first sprint to stand up the deployment pipeline against a dummy model. Make sure the model can be promoted, monitored, and rolled back before the real model exists. You will catch every wiring issue when there is no sponsor pressure.
4. Run a canary, not a launch
“Go-live” should not mean “all users on day one.” The right pattern is to canary the model to 5% of users (or 5% of the workflow), measure the production KPI for two to four weeks, and then ramp. If the KPI does not move, you have cheap, clean evidence — not an embarrassing rollback.
Build the cohort routing into the system from the start. A/B testing infrastructure for AI is not optional; it is the only way to prove that the model is doing what you said it would.
5. Make oversight a feature, not a meeting
Every model has edge cases that need human review. Build the human-review workflow as a first-class product surface — a queue, an interface, a feedback mechanism that flows back into the training data. If the only review is a monthly steering-committee meeting, the model is not being supervised; it is being audited too late.
The byproduct of a good human-review workflow is a continuously growing dataset of labelled hard cases — which is the highest-value training data your team will ever produce.
6. Plan the operating model before launch, not after
The single most common cause of post-launch model death is: nobody owns it. The data scientist who built it moved teams. The engineer who deployed it left the company. The business owner thinks the engineer maintains it; the engineer thinks the business owner does. The model drifts, no one notices, until a customer notices for them.
Before launch, write down the operating model: who monitors what dashboard, on what cadence, with what alert thresholds, escalating to whom, with what budget for retraining, with what authority to roll back. If you cannot answer these questions, you are not ready to ship.
7. Build the second use case in parallel
Single-pilot AI programs almost never survive. The political cost of the first pilot is high; if it does not produce a second use case quickly, the organisational memory of the AI investment fades. The teams we see succeed pick a portfolio of 3–5 use cases at the start, sequence them so that one finishes every quarter, and rotate the team across them so that learnings compound.
This also creates the political condition for an AI Centre of Excellence — which is the only structure we have seen that scales past a handful of models without descending into chaos.
The honest assessment for executives
If your AI program has been running for more than 12 months and has not yet produced a single model running in production against a measurable business KPI, the probability you fix it by doubling down on the same approach is low. Stop the next pilot. Run a candid retrospective. Pick a smaller, less glamorous use case where the data is already trustworthy and the KPI is already on a dashboard someone reads. Ship that one cleanly.
AI programs that survive look boring from the outside. They focus on a few high-confidence wins, instrument them ruthlessly, and grow the operating model as the portfolio expands. Programs that die look exciting in the slide deck and empty in the dashboard six months later.
The 70% failure rate is not a fact about AI. It is a fact about how AI gets bought, sold, and executed. The technical work is the easy part. The organisational work — KPIs, ownership, oversight, sequencing — is what separates the survivors.