Skip to content
Novandri.

Why AI Adoption Is Harder Than Building an AI Model

The model is the easy part. Read the evidence on why adoption stalls: uneven productivity gains, hidden technical debt, and the cost of verifying output.

Rachmadwipa Novandri10 min read

Teams do not fail at AI because the model was hard. They fail because the model worked and the organisation did not change. That distinction sounds obvious until you look at the numbers, and the numbers are not what most people expect: adoption has been unusually fast, the measured productivity gains are real but small, and the largest measured effect so far is that experienced developers got slower.

Direct answer:

Building a working model is the least risky part of AI adoption.

What follows — deciding where the tool fits the workflow, measuring whether it actually helps, and paying for the surrounding system — is where the difficulty lives. Three bodies of evidence explain the gap: hidden technical debt makes production systems much larger than the model; measured productivity gains are unevenly distributed and currently small in aggregate; and where a tool's output must be verified by someone who already knows the answer, the verification cost can exceed the saving.

34%improvement for novice and low-skilled workers in the customer-support studyBrynjolfsson, Li and Raymond, 2024
19%longer to complete issues for experienced developers using AI toolsMETR, 2025
1.4 percentof total work hours in reported time savingsBrynjolfsson, Li and Raymond, 2024
A four-step flow: build the model, map the workflow, verify output, then change the organisation.
Conceptual: adoption is an organisational change. Teams fail because the model worked and the organisation did not change.

The model is a small part of the system

The clearest account of why AI projects stall after the model works comes from Sculley et al., whose paper is titled Hidden Technical Debt in Machine Learning Systems. Its argument is structural rather than technical: the model is a small, replaceable component inside a much larger system that includes data collection, feature extraction, serving infrastructure, configuration, and monitoring — and the parts that are easy to overlook are the parts that fail.

This reframes the plan. "Improve the model" is the lever with the shortest feedback loop and the least leverage on the outcome. The levers that determine whether the system works in production are unglamorous: can you serve it reliably, can you tell when it has degraded, can you roll it back, and does anyone own the pipeline when it breaks at 03:00.

The practical consequence is that adoption budgets are usually misallocated. Money goes into the modelling work because that is where the demo lives, and the funding for the parts that determine durability is left to be discovered later, when it is no longer optional.

Adoption has been fast — which is not the same as useful

The most surprising evidence for a company planning an AI rollout is how quickly the technology spread.

Brynjolfsson, Li and Raymond report the results of "a series of nationally representative U.S. surveys of generative AI use at work and at home". As of late 2024, "nearly 40 percent of the U.S. population age 18-64 uses generative AI". Among employed respondents, "23 percent had used generative AI for work at least once in the previous week, and 9 percent used it every work day". Relative to each technology's first mass-market product launch, "work adoption of generative AI has been as fast as the personal computer (PC), and overall adoption has been faster than either PCs or the internet". Generative AI and PCs also showed "very similar early adoption patterns by education, occupation, and other characteristics".

So the problem is not that people will not use it. On this evidence, adoption is close to the fastest of any general-purpose technology we have measured.

The problem is what the use is worth. The same paper reports that "Between 1 and 5 percent of all work hours are currently assisted by generative AI, and respondents report time savings equivalent to 1.4 percent of total work hours." The authors' conclusion is deliberately careful: this "suggests that substantial productivity gains from generative AI are possible" — the potential is there, but it has not been realised in the aggregate numbers yet. A 1.4% aggregate time saving, in a workforce where a third of people touch the tool weekly, is a small number, and it is the number an organisation's aggregate productivity will actually move by.

This is the gap that matters for adoption planning. The tool is everywhere; the measured value from it is thin. A rollout plan that treats ubiquity as proof of value will over-project.

Where the gains land, and where they do not

The other half of the picture is who benefits. In their customer-support study of 5,179 agents, the same research group found productivity measured by issues resolved per hour rose "by 14% on average, including a 34% improvement for novice and low-skilled workers but with minimal impact on experienced and highly skilled workers".

The pattern is consistent with the trial evidence from METR, which points the other way for experienced practitioners. METR randomised sixteen experienced developers working in large open-source repositories — "averaging 22k+ stars and 1M+ lines of code" — to use AI coding tools, and measured 246 completed real issues. The result was that "When developers are allowed to use AI tools, they take 19% longer to complete issues — a significant slowdown that goes against developer beliefs and expert forecasts."

Read that carefully, including the authors' own caveat: "You only had 16 developers, so these results will not generalize/replicate." Sixteen developers is a small sample; 246 issues gave the study enough power to reject the null of no effect. So the honest conclusion is not "AI slows developers down" but something narrower and more useful: for a developer who already knows the codebase well, generating and reviewing an assistant's patch cost more than writing it directly.

Put the two findings together and a coherent mechanism emerges. The tool is most valuable when the user does not already know the answer — it supplies knowledge or a starting point they lacked. It is least valuable, and possibly negative, when the user already holds a strong prior, because then the output still has to be read, checked, and corrected, and that cost does not vanish. The 34% gain for novice workers and the 19% slowdown for experienced ones are not contradictory findings; they are the same finding seen from two ends of the expertise distribution.

That mechanism is an interpretation, and it should be treated as a hypothesis to test on your own team rather than a law. But it is a far more actionable hypothesis than "AI is a productivity tool" — it tells you to segment your users by prior knowledge before you roll anything out, and to expect the curve to be downward for your most senior people.

A test you can run before you roll anything out

That discovery can be done in weeks, and it belongs before the rollout rather than after it.

  1. Segment by prior knowledge, not by seniority. Use the heterogeneity finding directly. Run the same task with high-experience and low-experience users and compare. If the effect is flat, the tool is a training aid; if it is negative for the experts, restricting it to them is a decision to make deliberately.
  2. Measure the baseline first. If you do not know the current cycle time, error rate, or throughput, you cannot attribute a change to the tool. The METR design — randomised, on real work, with a pre-registered outcome — is a reasonable template.
  3. Count the verification cost. For tasks with a knowledgeable reviewer, time the review, not the generation. If review time equals or exceeds the time the tool claims to save, the workflow has not improved even if the output looks better.
  4. Instrument the system, not just the model. Latency, error rate, and a distribution-shift signal. Sculley et al.'s point is that these are the components that fail, and monitoring is not an optional extra in production.
  5. Budget for the parts that are not the model. Data pipelines, serving, evaluation, rollback. If the business case assumes the model is the cost, it is wrong by an order of magnitude.

What to actually do with this

The useful conclusion is not pessimistic. Adoption is fast, which means the tools will be used whether or not your organisation designs for it; the interesting work is designing where they sit.

That means segmenting your users by prior knowledge and expecting different results for each group; measuring outcomes rather than counting seats; and treating the non-model parts of the system as the actual project. The teams that get value from this wave will be the ones whose bottleneck is knowledge transfer — because that is where the measured 34% is — and not the ones whose bottleneck is expert execution speed.

For the economic framing of the same question, see The Economics of AI: Why Intelligence Is Becoming a Commodity, and for the pipeline mechanics, see MLOps: From Experiment to Production Pipeline.

A worked example

The following is hypothetical and illustrative. Imagine a team planning to roll out an AI coding assistant and expecting a uniform productivity gain. The evidence suggests testing that first. The first check is segmentation by prior knowledge rather than seniority: run the same task with high- and low-experience users, because the customer-support finding shows gains concentrated among novices (34%) while experienced workers saw minimal impact, and METR found experienced developers took 19% longer. The second check is a measured baseline: record current cycle time and randomize on real issues, following the METR design of sixteen developers and 246 issues, so a change can be attributed to the tool rather than to optimism. The third check is verification cost: time the review of an assistant's patch, not its generation, and compare that with the claimed saving. The outcome is bounded: the team offers the tool as a training aid for less experienced staff, where the gain is documented, and does not mandate it for experts whose review time cancels the benefit, while budgeting for the serving and monitoring parts the model alone does not cover.

A counterargument

A critic could argue that the article draws sweeping organisational conclusions from measurements that are narrow and early. The METR result rests on sixteen developers, and the authors themselves say those results will not generalize or replicate; the 1.4 percent aggregate time saving came from surveys taken while adoption was still accelerating.

Brynjolfsson, Li and Raymond conclude that substantial productivity gains are possible, so treating the current thin aggregate as a ceiling risks mistaking a snapshot for the steady state.

The piece acknowledges both caveats and calls its expertise mechanism a hypothesis, yet the practical advice - segment users by prior knowledge and expect senior people to slow down - could prematurely cap investment in expert workflows before better tools and redesigned processes change the picture.

References & Further Reading