AI proof of concept: how to run one that reaches production

Why so many AI proofs of concept stall

An AI proof of concept is easy to make look good. A capable model, a few hand-picked examples and a polished demo will impress a meeting room. Whether the same feature works on your real data, at your volume, at a cost you can carry, is a different question, and it is the one that decides whether the project reaches production.

The research firms that track this are blunt about the gap:

• In July 2024 Gartner predicted that “at least 30% of generative AI (GenAI) projects will be abandoned after proof of concept by the end of 2025”, due to “poor data quality, inadequate risk controls, escalating costs or unclear business value”. In March 2026 it reported that by the end of 2025, “over 50% of GenAI projects were abandoned after the proof-of-concept stage”.
• In June 2025 it predicted that “over 40% of agentic AI projects will be canceled by the end of 2027”, for the same mix of reasons: escalating costs, unclear business value and inadequate risk controls.
• A 2024 RAND study, based on interviews with 65 experienced data scientists and engineers, notes that “by some estimates, more than 80 percent of AI projects fail” and names five root causes. The first is that stakeholders “misunderstand, or miscommunicate, what problem needs to be solved using AI”; the second, that the organisation “lacks the necessary data”.

These are forecasts and interview studies, not audits of every project, so treat the exact percentages with care. (You may also have seen “95% of AI pilots fail”. That figure comes from a non-peer-reviewed 2025 working report whose method has been widely criticised, so we don’t rely on it.) The pattern behind them is consistent, though: projects rarely stall because the model can’t do the task. They stall because the problem was vague, the data wasn’t ready, nobody defined what “good enough” meant, or the running cost was discovered too late.

A useful proof of concept is designed to answer those questions early, cheaply and honestly.

Pick a use case you can measure

The best first candidates share a few traits:

• Volume. The task happens hundreds or thousands of times a month, so small gains add up and there is plenty of history to test against.
• Examples exist. Past tickets, invoices, emails or documents, with the outcome a person chose, give you a ready-made answer key.
• Errors are recoverable. A person can review uncertain cases, and a wrong answer costs minutes, not money or safety.
• A baseline exists. You know roughly how long the task takes today and how often it goes wrong.
• One owner. Someone in the business owns the process and can say whether the result is good enough.

Ticket triage, invoice and document extraction, lead research and answering questions from internal documentation usually fit. “Use AI across the business”, rare high-stakes decisions, and processes with no historical data usually don’t, at least not as a first project.

Write the success criteria before you build

Decide what “working” means before anyone writes a prompt. Anthropic’s own guidance for building with its models starts at the same point: “Building a successful LLM-based application starts with clearly defining your success criteria and then designing evaluations to measure performance against them.” Good criteria, it adds, are specific, measurable, achievable and relevant. Criteria written after the demo tend to describe the demo.

The thresholds above are placeholders to show the shape; yours come from the baseline. The point is that each one is a number someone agreed to in advance, so the go or no-go decision at the end is a comparison, not a debate.

Build the evaluation set first

An evaluation set is a collection of real inputs with the answers a competent person would give: the tickets and where they should have gone, the invoices and the fields that should have been extracted. It is the single most useful thing a proof of concept produces, because it outlives the prototype. Every later change of prompt, model or provider can be tested against it in minutes.

• Use real, recent examples, with personal data removed or masked where it isn’t needed.
• Cover the hard cases on purpose: ambiguous requests, poor scans, unusual formats, the edge cases staff already know about.
• Hold some back. Keep a slice the team never looks at while iterating, and use it only for the final check, so you measure the system and not your tuning.
• Record why each expected answer is right, so disagreements are about the rule, not the example.

It doesn’t need to be large to be useful. Writing about agents, Anthropic’s engineers suggest that “20-50 simple tasks drawn from real failures is a great start”, and warn that “evals get harder to build the longer you wait.” OpenAI’s advice is the same in fewer words: “Evaluate early and often.”

A four-week plan

A focused proof of concept for one use case fits in two to four weeks. A typical shape:

Start simple on purpose. Anthropic’s engineering guidance on agents recommends “finding the simplest solution possible, and only increasing complexity when needed”, noting that “for many applications, however, optimizing single LLM calls with retrieval and in-context examples is usually enough.” A multi-step agent is worth its extra cost and failure modes only when a simpler workflow demonstrably falls short.

This is the shape of our own AI agents and automation engagements: usually a two-to-four-week assessment and working prototype, then iterative delivery measured against the evaluation set.

What changes between the proof of concept and production

Passing the proof of concept means the idea works. Production adds everything that keeps it working:

Most of this is ordinary software engineering. It is also where the bulk of the cost sits, which is why a proof of concept should end with a production plan and a cost forecast, not just a demo. We break those costs down in what an AI agent costs to build and run.

Ownership matters too. In September 2026 Gartner predicted that by 2028, 70% of enterprises will abandon agentic AI built for them through vendor “forward-deployed engineering”, trapped by rising costs and unable to evolve the systems themselves. Whoever builds your proof of concept, make sure you own the code, the prompts and the evaluation set, and that your team can run them.

Go, iterate or stop

At the end, compare the results with the criteria you wrote in week one:

• Go when the system meets the thresholds on the held-back examples and in the shadow run, and the cost per item is comfortably below the value of the time it saves.
• Iterate when it misses on specific, understood failure types that more data, better retrieval or a narrower scope could fix. Set a short, fixed time box for the next round.
• Stop when it misses broadly, when the errors it makes are the expensive ones, or when the process itself turned out to be unclear. A stopped proof of concept that cost a few weeks is a good outcome next to a rollout that fails in front of customers.

Handle risk and compliance from day one

A proof of concept touches real data, so the basics apply from the first week:

• Check how the provider uses your data. OpenAI states that “data sent to the OpenAI API is not used to train or improve OpenAI models” unless you opt in; Anthropic’s commercial terms say it “may not train models on Customer Content from Services”; Google’s paid Gemini API tier says content is “not used to improve our products”, while its free tier is. Confirm the terms for the plan and region you use, and whether data is retained for abuse monitoring.
• Use a recognised framework as a checklist. NIST’s AI Risk Management Framework (2023) and its Generative AI Profile (2024) are voluntary, practical guides to governing, mapping, measuring and managing AI risks. NIST notes the framework is being revised.
• Know which rules apply. If you serve users in the EU, the AI Act has applied in stages since February 2025 and became generally applicable on 2 August 2026. After the 2026 “AI Omnibus” amendment, the rules for high-risk uses in sensitive areas apply from 2 December 2027. Check early whether your use case falls into one of the Act’s high-risk categories.
• Keep a decision log: which data was used, which model and version, what was tested and what was decided. It costs minutes and answers most of the questions an auditor, or a future team, will ask.

Sources

Every source below was read on 26 October 2026. Prices and product details change; check the current page before relying on a figure.

1. Gartner: Gartner Predicts 30% of Generative AI Projects Will Be Abandoned After Proof of Concept By End of 2025 (wording checked via reprints)
2. Gartner: Post on GenAI projects abandoned after proof of concept (11 March 2026)
3. Gartner: Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027 (wording checked via reprints)
4. Gartner: Gartner Predicts 70% of Enterprises Will Abandon Agentic AI Built by Vendor Forward-Deployed Engineering by 2028 (wording checked via a reprint)
5. RAND Corporation: The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed (RR-A2680-1; wording checked via reprints)
6. 80,000 Hours: The story behind the bad AI stat that moved markets and misled millions
7. Anthropic: Define success criteria and build evaluations
8. Anthropic: Demystifying evals for AI agents
9. Anthropic: Building effective agents
10. OpenAI: Evaluation best practices
11. OpenAI: Data controls in the OpenAI platform
12. Anthropic: Commercial Terms of Service
13. Google: Gemini Developer API pricing (data use by tier)
14. NIST: AI Risk Management Framework
15. NIST: AI RMF: Generative Artificial Intelligence Profile (NIST AI 600-1)
16. European Commission: AI Act
17. European Commission: AI Omnibus enters into force