Software Engineering UKSoftware Engineering UK
Stop Tracking LoC: Better AI Dev Metrics
Software Engineering UK

Stop Tracking LoC: Better AI Dev Metrics

Kelly MatthewsWords by Kelly Matthews

Further up the stream

The dashboard that proves an AI investment is working is usually the one that proves nothing at all. The received view treats AI productivity as a volume problem: more suggestions accepted, more pull requests merged, more generated code in the repository. The true position is narrower and harder. The only useful question is how much of that output survives review, reaches production and remains correct when the next change arrives.

The account CIO published sets out the gap from inside the industry. Budgets keep rising while technical leaders report unclear returns, and the reason sits in the lifecycle. A model produces code in seconds, and errors, missed internal requirements and rework consume the saving downstream. At Pearl, as the same account recounts, a request for pipeline automation came back as GitHub Actions workflows for an environment built on Azure DevOps and Jenkins: plausible, idiomatic, built for the wrong platform. The engineers who caught the mismatch rebuilt the integration. CIO calls output of that kind "convincing despite lacking important context". Review time climbs faster than task completion as generated volume grows, and the queue absorbs the gain.

The lesson is not that AI output is worthless. Output is an input to a delivery system, and the system's constraints decide the result. A team that counts generated volume without measuring the review queue, the rework and the survival of shipped changes is measuring its own optimism. Blind trust in a rising line count creates technical debt with a clean audit trail.

The saving arrives at the review queue

index.dev's ROI study found that individual output can rise while company-level delivery metrics stay flat because review, quality assurance and release cadence do not accelerate with drafting. The same study reports that review time can grow sharply and that experienced developers sometimes take longer with AI assistance. The tool speeds up the part of the work that was already fast. The queue is the new constraint.

getdx.com's analysis puts the same point in measurement terms. It found that AI-authored share climbed quickly, pull request size grew, maintainability improved slightly, and developers' confidence that a change would not break something fell. The direction of output and the direction of trust were not the same. A team can ship more code and feel less certain about what it shipped, and that gap is exactly what a line count hides.

dev.to's critique of agent metrics adds the accountability problem. An acceptance rate can be high because the agent is good, or because reviewers have stopped reading. Without review depth, rollback rate and maintenance attribution, the number measures the review process and leaves the output unexamined. That is the failure mode to design against before AI-generated code becomes the normal case in a repository.

A dashboard that counts generated configuration as output scores the wrong-platform workflow as productivity. A review process that checks platform fit scores it as rework. The second score is the one that protects the system.

A checklist that measures functional value

A useful scorecard for AI-assisted development has to answer several questions in the same review cycle. Collapsing them into one number destroys the information the exercise exists to produce. CIO draws the distinction that makes the scorecard necessary. Token usage and developer participation show how much a tool is used; delivery, effort and quality read together show whether the work is getting better. The entries below form a scorecard to place in front of an engineering leadership team, and each one needs a baseline before it can support a decision.

Change survival asks what proportion of merged changes are still in production, unchanged, after a fixed interval. Track it separately for AI-assisted and human-authored changes, and set it beside the rollback rate. Tek Ninjas calls this change durability and argues that high throughput with low durability is churn dressed as velocity. A change that was merged and then reverted within a month is a production cost, whatever the pull request looked like on the day it landed.

Review depth counts substantive comments, requested changes and rounds of review per pull request. Approval counts as the closing event in that sequence, and the depth sits in the steps that precede it. dev.to's analysis makes the point that a first-pass approval with no comments can mean the change was trivial or that nobody read it. Segment the number by origin and by risk tier, and watch for review effort on AI-assisted changes falling while their volume rises. That divergence is a leading indicator of rubber-stamping.

Maintenance attribution follows the follow-up work. When a defect fix or refactor touches code, record who or what wrote the original change, then compare the share of follow-up work attributable to AI-assisted and human-authored code. A disproportionate share is the signal that the tool is producing code that is cheap to generate and expensive to own. This is the metric that connects today's merge to next quarter's maintenance burden.

Flow with a quality counterweight pairs cycle time with change failure rate and rework. Shortening the time from first commit to production is welcome only if failure and rework hold steady or improve. index.dev's study found that individual throughput can rise while company-level delivery metrics stay flat because downstream processes do not move at the same speed. A team that ships faster and fixes more is not faster. It has moved the queue.

Context reuse asks whether an agent builds on earlier decisions or starts fresh on each task. It works as a leading signal, and its job ends there. Repeated architecture arguments in review, or the same convention being relearned in every pull request, point to a context problem upstream of the code. dev.to's critique treats this as speculative but useful, and that is the right status: track it and keep it out of performance targets.

Instrument the workflow where the work happens

The instrumentation has to serve the workflow. Tool vendor reporting measures its own product. CIO describes the arrangement that survives contact with production as human judgement on risk and complexity with automation on the routine cases. Lower-risk changes such as unit tests pass through AI-assisted review, and production-facing code still needs a person. CIO argues that the workflow itself should be rebuilt around what the tool can do, with a baseline for development speed and software quality taken before the rollout expands. Automate the collection, and keep the decision with a named reviewer.

The AIRC's Measure function says what should be measured depends on the purpose, audience and needs of the evaluation, and that organisations should document risks or trustworthiness characteristics they will not measure, with justification. That sounds bureaucratic until a team tries to compare an AI-assisted pull request with a human one. Without a documented choice of metric, the comparison defaults to volume.

Tek Ninjas recommends keeping DORA as context, running new measures as observational only for a full quarter, and validating each one against outcomes before it becomes a target. The sequence matters because a metric that becomes a target before it has a baseline will be gamed by well-meaning engineers. A trend with a baseline behind it supports a decision. A single impressive number stands alone and falls alone. CIO's account of a falling engineering cycle time earns its place for the same reason: the number comes with a baseline, and the baseline is what makes it readable.

The query below produces change survival and review depth from pull request, review and revert records. The comment threshold is a judgement call, so document it beside the metric.

WITH merged AS (
  SELECT
    pr.id,
    pr.merged_at,
    pr.risk_tier,
    pr.author_kind,
    COUNT(DISTINCT rv.id) AS review_rounds,
    SUM(CASE WHEN rv.state = 'CHANGES_REQUESTED' THEN 1 ELSE 0 END) AS changes_requested,
    SUM(CASE WHEN c.substantive THEN 1 ELSE 0 END) AS substantive_comments
  FROM pull_requests pr
  LEFT JOIN reviews rv ON rv.pr_id = pr.id
  LEFT JOIN comments c ON c.pr_id = pr.id
  WHERE pr.merged_at IS NOT NULL
  GROUP BY pr.id, pr.merged_at, pr.risk_tier, pr.author_kind
),
survival AS (
  SELECT
    m.*,
    CASE WHEN r.reverted_at IS NULL THEN 1 ELSE 0 END AS survived_30d
  FROM merged m
  LEFT JOIN reverts r
    ON r.pr_id = m.id
   AND r.reverted_at <= m.merged_at + INTERVAL '30 days'
)
SELECT
  author_kind,
  risk_tier,
  COUNT(*) AS merged_prs,
  AVG(survived_30d) AS survival_rate_30d,
  AVG(review_rounds) AS avg_review_rounds,
  AVG(changes_requested) AS avg_changes_requested,
  AVG(substantive_comments) AS avg_substantive_comments
FROM survival
GROUP BY author_kind, risk_tier
ORDER BY risk_tier, author_kind;

Change survival and review depth sit in the same output, segmented by risk tier and by who wrote the change. The substantive-comment flag is where the judgement lives, so write the definition down and keep it with the query.

None of this needs a new platform. Pull request, review and revert records already hold the evidence. The work is to define the thresholds, baseline them, and read the numbers together before the next AI rollout resets the queue. A line count can stay in the repository as a size signal. It has no business on a productivity dashboard.

Sources

Source: The illusion of AI productivity in software development and what to measure instead, CIO

The usual questions

Why is tracking lines of code misleading for AI?

Volume does not equal value. AI can produce large amounts of code that lack context or fit the wrong platform, increasing the review burden without improving delivery.

What is change durability in AI development?

It is the proportion of merged changes that remain in production unchanged after a fixed interval. Low durability suggests that high velocity is actually just churn.

How does maintenance attribution help track AI quality?

It records whether follow-up fixes or refactors touch AI-authored or human-authored code. This reveals if AI code is cheap to generate but expensive to maintain.

Filed so far