Skip to main content
onext technology
AI 20 November 2026 - 7 min read

AI agent pilot: why your team picks the easy project

If the pilot runs on a new project with volunteers, your team learns little about how the agent will do in its everyday code.

Bernat López
Founder and CEO of onext
Two developers in an open-plan office review code on a monitor: one, in profile, points at the screen while the other, seen from behind, rests his hand on his chin; behind them, a colleague holding a mug looks at a whiteboard with diagrams

"Let's run the pilot on the new service, there's no legacy code there." It sounds sensible. The service is small and two people who already used a coding agent on their own run it. In a month there will be a demo for the leadership team. The pilot goes well. Then the agent reaches the main product, the one the rest of the team maintains, and nobody knows what to expect.

I'd call it a showcase pilot: a pilot designed to go well and, for that very reason, of little use for deciding anything. It measures what the agent does on the ground where it is needed least, with new code and people who are already convinced, and leaves untested the code the team maintains every week.

The thesis

I am not saying that a pilot on a new project is useless; I am saying it cannot tell you whether the agent will work in the rest of your code. The biggest published gains come from new, short tasks, and they shrink when the code is old and complex. A useful pilot runs on a real enhancement to the product, a change the business has already asked for, in the code that supports it, with its rules and its tests. And it writes down before it starts which result would mean carrying on and which would mean stopping.

In Track M the pilot runs on a real enhancement, to measure the improvement against a baseline. And we do it in two stages: first we do it, with the team watching, and then the team does it, with us watching, so that we make sure the process is handed over correctly.

The showcase pilot answers another question: whether the tool works in good conditions. That question almost always gets a yes, and that is why it teaches so little to whoever has to decide.

What the data says

The figure that turns up most in presentations and those from later studies measure different things. The difference lies in the type of task.

  • Sida Peng, Eirini Kalliamvakou, Peter Cihon and Mert Demirer (GitHub, MIT and Microsoft Research, 2023) asked 95 professional programmers to implement an HTTP server in JavaScript as quickly as possible. Those with the assistant finished 55.8% faster. It was a new task, with no existing code.
  • Elise Paradis and her team at Google (2024) ran a controlled trial with 96 full-time engineers on a complex, enterprise-grade task. Their best estimate is about 21% less time, with a wide confidence interval.
  • Kevin Zheyuan Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng and Tobias Salz studied three experiments at Microsoft, Accenture and a large electronics manufacturer. On average, weekly completed tasks rose by 26%. For recent hires and less experienced developers the rise was 27% to 39%, and for the most senior, 8% to 13%. After a year, average adoption was about 60%. MIT Sloan reported it in November 2024.
  • Yegor Denisov-Blanch, a Stanford researcher, presented in 2025 a matrix based on 136 teams at 27 companies: indicative gains of 30 to 40% in new, simple code, and of 0 to 10% in old, complex code.

Peng and Cui include co-authors from Microsoft or GitHub, and the Paradis trial is run by Google on its own tools. The Denisov-Blanch matrix comes from a talk, with no published paper or method, and he presents it himself as indicative. Even so, Peng, Paradis and that matrix point the same way: the measured gain drops the more the task looks like the daily work of a team with its own code.

What it looks like in the team

  • The pilot goes into the microservice that shipped last month and that no customer uses yet.
  • The two people on the pilot already used an agent at home and volunteered on day one.
  • The demo shows a feature built in an afternoon, and nobody asks how long reviewing it took.

Why it is understandable

Whoever organises the pilot has something at stake. They persuaded the leadership team to fund it and need to come back with a result to show. Choosing the easy ground lowers the risk of a visible failure.

Leadership has its share. If it asks for results in a month and penalises the pilot that turns out so-so, the team learns to pick the project that cannot go wrong.

What a pilot that does not decide costs

The first cost is the decision. With the result of that pilot, the company buys licences for the whole department or brings the agent into the main product, and finds out in the legacy code what the pilot never tested. Taking the practice from one team to the whole department is hard enough when the first team really works. If the starting point is an inflated result, you start with the wrong expectation.

The second cost is what nobody learns. The hard part of bringing agents into everyday code is preparing the ground: writing down the team's rules, explaining the product in the repository's context and having tests that warn you when something breaks. A small new project needs almost none of that, so the pilot ends without anyone having done it.

What to do

The people who do the work (the team that will use the agent):

  • Choose for the pilot a real enhancement to the product that touches code with more than a year of history.
  • From day one, write one line per task: whether the agent helped and how long it took to review what it delivered.
  • Before the first task, put the team's rules in the repository, along with the safety net of tests every change has to pass.

The people who lead:

  • Write down before starting which result would mean carrying on and which would mean stopping, with a figure and a date.
  • Include at least one team that did not volunteer and that maintains legacy code.
  • Count as done only the change that passes the repository's automated checks, and measure what you will later measure across the whole department: for example, the time from a change being requested to it reaching production, and how many changes have to be redone.

Before you choose the project

At our workshops I have seen and heard the same question many times: whether using generative AI is easier on new projects. I think the opposite: it brings far more value and improvement on advanced projects. On old code agents are especially good, and that is because it is much harder for a human to follow spaghetti code than for an AI. For me, the first thing generative AI was ready for was helping with legacy code. In the Thoughtworks radar, "Using GenAI to understand legacy codebases" moved to Adopt in November 2025, for understanding that code.

The studies measure less speed gain on old code, but that is where the daily work is and what is hardest to understand: that is why the pilot has to be there.

A showcase pilot tells you the agent works where you already knew it would. The pilot worth running happens in the code your team maintains every week, with the result that would mean stopping written down before it starts.

Frequently asked questions

What is a showcase pilot?

It is a coding agent pilot designed to go well, on a new project and with volunteers. It shows that the tool works in good conditions and leaves untested the code the team maintains every week.

Why does a pilot on a new project give better results?

Because new, short tasks are the ones that gain most from a coding assistant. In a 2023 trial with a task like that, the group with the assistant finished 55.8% faster. In a 2024 Google trial with a complex enterprise task, the best estimate was about 21% less time.

Where should the pilot run?

On a real enhancement to the product: a change the business has already asked for, made in the code that supports it, with its rules and its tests.

Should the pilot use volunteers?

They can take part, but it helps to include at least one team that did not volunteer. In the experiments by Cui and his co-authors, average adoption was about 60% after a year, and a pilot with volunteers only hides that part.

What should be decided before the pilot starts?

Which result would mean carrying on and which would mean stopping, with a figure and a date, and what will be measured. It helps to measure what you will later measure across the whole department, such as the time from a change being requested to it reaching production.

How much do agents improve productivity in legacy code?

There are few figures and none of them applies to every team. The matrix Yegor Denisov-Blanch (Stanford) presented in 2025 points to indicative gains of 0 to 10% in old, complex code, and it comes from a talk with no published paper. That is why it is worth measuring in your own code.

Sources

Bernat López
Written by
Bernat López
Founder and CEO of onext

Bernat López is founder and CEO of onext, an AI boutique. He helps development and product teams work with AI with method —specification before building, automated checks and Spec-Driven Development— and applies to his own company what he proposes: onext runs on its own agentic system.

LinkedIn →

Which code will you test the agent on?

Let's talk about your next pilot and how to set it up so it helps you decide.

See the free online workshop

What we build stays with your team.