An organization can use more AI without becoming better at delivering software. More sessions might reflect wider adoption, repeated attempts, or a growing amount of unfinished work. Understanding the value requires looking beyond the activity to what changed for the team and the people it serves.
Improvement begins with a purpose
An investment in agentic work might aim to shorten the wait for a customer fix, reduce recurring maintenance effort, or make a modernization project affordable. Each purpose calls for a different assessment. The most frequently used capability isn't necessarily the one contributing most to those goals.
Some value appears before a change is shipped. An investigation may rule out an expensive approach or help a team understand an unfamiliar system. Measuring every activity by whether it creates a merge request would miss that result and could encourage unnecessary changes.
What the team measures should reflect the work the organization values. A completed customer improvement, a resolved maintenance need, or a decision made with less uncertainty can be more meaningful than an agent session. The team needs a way to recognize the result without assuming that the agent's completion status establishes it.
Faster work and financial savings aren't interchangeable
Reducing implementation effort can free capacity, but the organization still has to decide how to use it. That may mean tackling work that previously waited, improving quality, or reducing dependence on overtime. Those benefits are real possibilities without being automatic reductions in expenditure.
Financial return needs an explanation of how the benefit becomes financial value. Converting every reported hour saved into money can overstate the result if the time is fragmented, absorbed by review, or never redeployed. The same benefit can also be counted twice if increased delivery and saved effort are valued independently when one produced the other.
Costs extend beyond the model call. Context preparation, evaluation, review, repair, infrastructure, and ongoing support all contribute to the investment. Cost per accepted change can help compare delivery efficiency, but it doesn't tell you whether the change improved anything for its users. The return comes from that improvement, not from the amount of AI used to produce it.
The explanation matters as much as the trend
A before-and-after comparison can help a team learn, but other things may have changed too: the kind of work, staffing, pipeline performance, or release process. Separating those effects makes the conclusion more credible than attributing every improvement to AI.
Measures also influence behavior. Individual targets for generated code, accepted suggestions, or completed agent tasks can reward activity while discouraging the investigation and collaboration that improve delivery. Team-level results and the experience of the people doing the work provide a better basis for deciding what to keep, change, or stop.
Further Reading
- Choosing where AI can improve delivery helps identify an opportunity and compare ways to address it.
- Evaluating business value and the cost of delivery develops an approach to baselines, comparisons, full costs, and investment decisions.