Measuring AI ROI: Getting Out of Pilot Purgatory

Many enterprises can report that AI helps. Far fewer can show where the money moved. In IBM's 2025 survey of 2,000 CEOs, they reported that 25% of AI initiatives had delivered expected ROI and 16% had scaled enterprise-wide, while 68% reported clear metrics to measure innovation ROI (not AI-specific) ^[1].

Our thesis, as NKK's view: pilot purgatory is mostly a specification and measurement problem. The evidence shows that gains depend on task and skill and that measurement is weak; it does not show where every stall starts. A pilot stalls when nobody specified, before the first prompt, the baseline, the owner, the route to cash and the evidence that would end it.

The pilot trap is a measurement problem before it is a technology problem

Headline percentages on what works look contradictory because they measure different things. BCG's 2025 study put 5% of companies at "future-built", 35% scaling and 60% "reaping hardly any material value" ^[4]; its 2026 release reports 7.5% and 41% for the first two groups ^[5]. The 2026 release does not say whether the two editions use matching samples or definitions, so the change is not a like-for-like trend. Wharton and GBK found 72% of leaders formally measuring Gen AI ROI and about three in four reporting positive returns ^[6]. Deloitte found 15% of generative-AI users reporting significant, measurable ROI ^[7].

Those are different thresholds (expected ROI, any positive return, P&L-visible value at scale), populations and evidence. None is audited financials, and "future-built" is BCG's own classification ^[5]. We therefore offer no failure rate.

The more useful observation sits inside the IBM release: 68% of CEOs report clear metrics for innovation ROI, yet they report only 25% of AI initiatives delivering expected ROI ^[1]. The units differ (CEOs versus initiatives, innovation versus AI), so this is not a like-for-like gap. The problem is mostly design. A pilot scoped to show the tool works (usage, satisfaction, demos) produces evidence of adoption. A CFO needs a baseline, a counterfactual and a cash path. Pilots built without those can neither graduate nor be cleanly stopped.

What the field evidence actually shows

The strongest evidence comes from field experiments, and it is positive but conditional. In a staggered rollout to 5,172 customer-support agents, issues resolved per hour rose about 15%; less experienced agents gained most, while the most experienced saw small speed gains and small quality declines ^[8]. Randomised trials at Microsoft, Accenture and a Fortune 100 company covering 4,867 developers found a 26% increase in completed tasks, with a standard error of 10.3% ^[9]; they ran inside participating companies. A six-month experiment with 66 firms and 7,137 knowledge workers found that the 80% of treated workers who used the tool spent about two fewer hours on email per week in the second half of the experiment; the authors detected no shift in the quantity or composition of tasks ^[10]; NBER discloses that at least one author worked at Microsoft.

Then the counter-evidence. METR's trial covered 246 tasks in mature projects and found that allowing early-2025 tools increased completion time by 19% ^[3]. Its February 2026 follow-up estimated an 18% speedup for a 10-developer returning subset (confidence interval from a 38% speedup to a 9% slowdown), but METR calls its data "only very weak evidence for the size of this increase" in speedup since early 2025 ^[11]. BCG's experiment with 758 consultants found GPT-4 users' performance 40% higher on a creative task inside the model's "frontier of competence" and 23% worse on a problem-solving task outside it ^[12].

Stanford's 2026 AI Index summarises the pattern: "Productivity gains from AI are largest in structured, measurable work where outputs are easy to monitor." ^[13]

What this evidence does not show: none of these studies measures enterprise profit. They measure task output, mostly with 2023 to 2025 tools. Gains vary by task, skill level and context, so measure per task type and worker segment, not per tool. A blended "AI productivity" figure averages novice gains and expert losses into a number that describes nobody.

Why a measured gain does not become a P&L gain

Task gains are necessary but not sufficient. Humlum and Vestergaard linked adoption surveys of 25,000 workers in 11 exposed occupations to Danish employer-employee records. Users reported saving 2.8% of work hours in surveys, yet the linked records showed no significant effect on earnings or recorded hours in any occupation. Pooled estimates ruled out effects above 1% and occupation-level estimates generally ruled out changes above 6%; only 3% to 7% of workers' productivity gains passed through to earnings. These are May 2025 figures; the March 2026 revision is retitled "Still Waters, Rapid Currents", rules out effects above 2%, and stresses task reorganisation ^[14].

Self-reports also run high. US users reported saving 5.4% of work hours, and the St. Louis Fed's summary cautions that workers may use saved time to ease up rather than produce more ^[15]. Among nearly 6,000 executives in four countries, about nine in ten reported no impact on employment or productivity at their firm over three years ^[16], while a survey of about 750 executives found perceived gains larger than measured ones, which the authors link to delayed revenue realisation ^[17].

Part of the lag is measurement. Brynjolfsson, Rock and Syverson argue that general-purpose technologies look unproductive early because complementary intangible investment is poorly measured ^[18]. That is an argument about measurement, not proof that AI gains exist.

We propose a simple model for the step most pilots miss, drawing on the pass-through gap in ^[14], the self-report gap in ^[15] and the intangibles lag in ^[18]. In the NKK Value Conversion Chain, activity becomes capacity, capacity becomes output, output becomes outcome and outcome becomes P&L. Apply it by writing, for each link, who decides what happens next, when you should expect to see it, and which guardrail metric must not worsen. The break is usually at capacity: freed hours turn into slack unless someone decides in advance to redeploy them to a queue where demand exceeds supply, or to remove them through attrition, contractor spend or hiring avoided. We call that the conversion decision. It does not hold for every use case, because quality gains can reach the outcome link without freeing capacity.

The NKK Value Tree: from the P&L down to a metric you can own

Start from the P&L and work down, not from the tool and work up. The NKK Value Tree has three branches under enterprise value: cost-out, revenue-up, and risk and loss avoided. Capacity is not a branch, because freed hours are not value until converted. Every leaf must name an owner, a baseline, a downstream outcome metric (never hours freed), a conversion mechanism (remove the cost, or redeploy to a named queue where demand exceeds supply), an evidence rung and a time to signal. It builds on Deloitte's benefit categories ^[2], McKinsey survey data reproduced by Stanford (self-reported; cost savings most often tied to software engineering and manufacturing, revenue gains to marketing and sales) ^[13], BCG's observation that leaders "measure the transformative impact directly in their P&L" ^[5], and FinOps unit economics ^[19].

Deloitte's data show why the tree is needed. In NKK's reading, efficiency benefits are reported far more often than revenue, and about 74% hope to grow revenue but only 20% are doing so: "Revenue growth largely remains an aspiration." ^[2]

To apply the tree: pick the branch, write the leaf, have finance sign the baseline and the conversion mechanism before launch, then review quarterly and scale or stop on pre-agreed thresholds. Where it breaks: branches overlap (a cost-out may also avoid risk); strategic uses such as building capability lack clean baselines, so log them as explicit strategic bets with a spend cap and a review date; and shared platforms need allocation rules, while FinOps flags a lack of generally accepted frameworks for cost allocation across multi-agent workloads ^[19].

Leading and lagging indicators, and the lag you should expect

Deloitte's EMEA survey found satisfactory ROI typically takes two to four years, against seven to twelve months as the typical payback for technology investments; only 6% reported payback within a year, and even among the most successful projects only 13% did ^[7]. Four in five Wharton and GBK respondents expect payoff in two to three years ^[6], but that is an expectation.

A single twelve-month ROI test will therefore stop projects that are on the curve and pass some that are not. We recommend reporting every link of the Conversion Chain together, each with its lead time and its paired guardrail, never one link alone. The lead times are heuristics to set per use case, and the chain assumes causal links that the Attribution Ladder must test.

Guardrails exist because volume metrics get gamed. NIST warns that "measurement approaches can be oversimplified, gamed, lack critical nuance, become relied upon in unexpected ways, or fail to account for differences in affected groups and contexts" ^[20]; it is a risk framework, not an ROI standard. KPMG's Q2 2026 pulse of 204 US leaders found 41% would consider "token-maxxing", rewarding token consumption with incentives and leaderboards, and KPMG warns of "incentivizing activity over outcomes" ^[21]. The SPACE paper makes the general point that developer productivity "cannot be measured by a single metric or dimension" ^[22].

The NKK Attribution Ladder: how hard can you attribute the result?

Attribution is the second half of measurement. Without a counterfactual, a rising metric may reflect seasonality, a concurrent reorganisation or simply the novelty of a new tool.

The NKK Attribution Ladder has six rungs. Rungs 0 and 3 to 5 map to cited studies; rungs 1 and 2 are standard practice (telemetry, before and after). Rung 0, self-report, is where many pilots stop, and it is unreliable: METR's developers estimated afterwards that AI had saved them 20% of their time, while the measured effect was 19% more time ^[3]. Rungs 3 to 5 use comparison groups: difference-in-differences with employer policy as the quasi-experiment ^[14], staggered rollout ^[8] and randomised assignment ^[9][10].

The rule: the size of the decision sets the rung. A team-level tool under a modest spend threshold can run on rung 2. Funding to scale, vendor commitments or headcount changes need rung 3 at minimum; default to rung 4 where waves are feasible.

Where it breaks. Holdouts can be ethically, contractually or operationally impossible, and treated and control workers contaminate each other when they share work. Most importantly, once a tool is the default, a without-AI control may no longer exist: METR's developers declined to take part or to submit tasks without AI, and METR judges its central estimate "likely a bad proxy" for real impact after listing several issues with the experiment ^[11].

Cost of ownership: measuring the denominator

ROI also fails at the denominator. KPMG's Q2 2026 pulse found 26% of leaders had full, real-time visibility into what their AI systems cost to operate, and 36% had direct token or usage controls ^[21]; the sample is 204 US leaders at organisations with $1 billion or more in revenue, so it is directional. The Q3 pulse (24 September 2026; 314 leaders, same revenue floor) reports 43% with usage or token budgets and 70% using AI monitoring dashboards, but did not repeat the visibility question ^[26]. Controls appear to be spreading; current real-time visibility is unmeasured. BCG reports corporate AI spend rising from about 1.7% of revenue in late 2025 to 3.3%, with more than 80% sitting outside enterprise IT ^[5]. An IT-budget view therefore understates cost.

Unit price is not total cost. Epoch AI found the price of reaching a given benchmark level fell between 9x and 900x per year depending on the milestone, and cautions that the fastest declines may not persist ^[23]. Vendor levers matter: one vendor lists cache hits at typically 10% of the standard input price and a 50% batch discount ^[24], though prices change. FinOps for AI proposes cost per inference, per token and per API call, and "Time to Achieve Business Value" ^[19]. These are run-layer measures. We recommend pricing the outcome: cost per resolved case, not cost per token.

The people layer is easy to under-budget. BCG's 10-20-70 heuristic puts about 70% of AI value in rethinking people, with 10% from algorithms and 20% from technology ^[25]. That ratio comes from BCG case work without a published method, so treat it as direction, not a rule. Our six-layer cost checklist, built on FinOps ^[19] and that ratio, is NKK's own layering and a checklist, not a fourth model.

What to do next

1. Pick three use cases and write a Value Tree leaf for each. Name the owner, metric, conversion mechanism and time to signal. Drop any use case that cannot terminate in a P&L line, unless you log it as an explicit strategic bet with a capped spend and a review date.
2. Fix the baseline before launch. Capture a trailing baseline with the same metric definition, and have finance sign it.
3. Set the evidence rung by decision size. Scale decisions need rung 3 at minimum; default to a staggered rollout (rung 4) where waves are feasible.
4. Make the conversion decision up front. State whether freed capacity will be redeployed or removed, and who decides.
5. Instrument cost per outcome. Capture the run layer automatically; estimate review time and change effort from the start.
6. Review at 90 days and re-baseline. Report every Conversion Chain link with its guardrail, and expect the P&L link to lag.
7. Agree stop and scale thresholds in advance. Record them in the launch document so ending a pilot is an evidence decision, not a political one.

Conclusion

In NKK's view, many pilots stall because no one specified the evidence that would end them. Define the metric, the baseline, the conversion and the stopping rule before the first prompt. The task gains are real where work is structured and the people are well matched to the tool, but they reach the P&L only through decisions someone owns. The NKK models are heuristics built on cited prior art; test them against your own data.