Why AI pilots stall
A pilot that runs for six weeks, demonstrates well, and then quietly stops being used is not a technology failure. It is a scoping failure, and it is an ordinary one.
Three things cause it. All of them are decided before any code is written.
Nobody defined “working”
Ask what success looks like and you will often get an answer like “it should summarise the tickets accurately”. That is not a definition anyone can test against. Accurate compared to what? Judged by whom? At what rate is it good enough to replace what happens today?
A usable definition looks more like this: given 200 real tickets from last quarter, the summary identifies the correct product area at least as often as a new starter does in their first month.
That version can be measured. It has a baseline — a new starter — so “good” is not an abstract standard but a comparison to what the business already tolerates. And it is written down before the build, so nobody gets to move the goalposts in either direction when the results arrive.
If you cannot write that sentence for your pilot, the pilot is not ready to start. That is a genuinely useful thing to discover in week one rather than week six.
The process was already broken
Automating a process does not fix it. It runs the same process faster and more consistently, including the parts that were wrong.
It is common to find approval workflows where several of the steps exist because of a system limitation that was resolved years ago and never revisited. Handing that to a model produces a faster version of unnecessary work, and it makes the unnecessary work harder to remove later, because now it is encoded in something nobody wants to touch.
Map the process before deciding to automate it. Quite often the answer is that four steps should become one, and once they do, the remaining step is simple enough that it does not need AI at all. That is a good outcome, even though it is a smaller project.
No one owned it after the demo
This is the one that kills pilots that were otherwise fine.
The build finishes, it works, everyone agrees it is useful, and then it degrades. The prompts stop matching how people actually write. An upstream system changes a field. Someone notices the output is wrong, works around it manually, and does not mention it. Six months later it is switched off.
An AI system needs an owner in the same way a report or an integration does: a named person who is responsible for noticing when it drifts, and who has the access and the time to correct it. Not a committee, and not “the vendor”.
If no one in your business will own it, the honest conclusion is that you should not build it yet. Buying a supported product is a reasonable alternative. Building something bespoke that nobody maintains is not.
What to do instead
Before committing to a build, get three things written down:
- A measurable definition of working, with a baseline drawn from how the work is done today.
- A map of the process as it actually runs, and a decision about which steps should exist at all.
- A named owner who will still be there in six months, with the access to keep it running.
None of that requires a vendor, a budget, or a technology decision. It usually takes a week or two, and it is the cheapest part of the whole exercise. It is also the part most often skipped.
If the answers do not come easily, that is information. A pilot that cannot be defined is not a pilot; it is an experiment with a deadline, and it will end the way those usually end.