Software Engineering UKSoftware Engineering UK
Is your code quality actually dropping?
Software Engineering UK

Is your code quality actually dropping?

Alexander HillWords by Alexander Hill

Further up the stream

Between 23 and 25 per cent of tests run without a person at the keyboard, a band that has barely shifted across two decades of tooling spend, and Diginomica's report on UiPath's testing business sets that number beside deployment rates five to ten times higher than a year earlier. The volume of code arriving for checking has changed beyond recognition over the same period. Those two facts carry most of the problem, and they explain why the question in the title resists a tidy answer: the property that fails first tends to be the one a team never instrumented.

Where the gap opens

Diginomica reports that AI coding agents have compressed delivery cycles while early evidence points to application quality slipping, and UiPath calls the resulting lag acceleration whiplash, a name for the delay that follows a burst of generation capacity. Generation capacity multiplied. Verification capacity did not.

Verification capacity has two components:

  • machine checks that run whether or not anybody is watching
  • human attention to judge what a machine cannot

The first is cheap to expand and dull to fund. The second is expensive, scales badly with headcount and is the first thing squeezed when release windows shorten.

The symptoms are already documented. UK Tech News reported in April on a ghost code problem in British teams, where coding velocity has risen while time-to-merge has stagnated, producing a culture of rubber-stamping and logic that runs today with no clear lineage, no human author and no long-term owner. Rubber-stamping is a symptom of arrival rate: it is what a review process does when changes arrive faster than a person can build a model of them.

What the usual instruments miss

Self-report will not catch the decline. SmarterArticles covers the METR randomised trial in which experienced open-source developers took 19 per cent longer to finish tasks with frontier assistants available while believing those assistants had made them 20 per cent faster. SmarterArticles also reports GitClear's analysis of 211 million changed lines, which found refactoring falling from a quarter of changed lines in 2021 to under a tenth in 2024 and new code revised within two weeks rising from 3.1 per cent in 2020 to 5.7 per cent in 2024.

The samples differ by orders of magnitude and the direction still instruct: the people closest to the work cannot reliably feel the difference between motion and progress. A dashboard of story points closed or pull requests merged will not settle it either, because volume is the quantity under suspicion.

Self-report is not the only instrument available, though. Three signals sit close to the work and cost little to collect. The share of changed lines rewritten within a fortnight of merge, the count of production incidents traced back to merges from the previous release and the number of review threads that reach approval within a minute of opening each act as a proxy for one thing: whether a reviewer built a model of the change before it landed. A team that watches those three numbers for a quarter learns more about its code health than any satisfaction survey can tell it.

Why review practice stalls at this arrival rate

The National Cyber Security Centre's guidance on producing clean and maintainable code makes the security case for an old point. Inconsistent, poorly laid out and undocumented code adds complexity to a system, and complexity hides bugs, some of which become vulnerabilities. An attacker needs one way in; a defender has to find all of them. Nothing in that argument depends on whether a person or a model wrote the line.

The remedies the NCSC sets out share a hidden assumption. Logical architecture, small and regular commits, peer review by two or more people, clear attribution of every change and documentation kept current as the system evolves all depend on human attention per change. Small commits are recommended because large ones are impractical to review, which is another way of saying that review cost is a function of diff size and arrival rate together. Raise the second and the first has to give somewhere.

Google's code review standard is candid about the trade-off. Its stated purpose is that overall code health improves over time, and its senior rule asks a reviewer to approve a change as soon as that change leaves the system healthier than it found it, with polish left as a suggestion. The guide wants reviewers to pursue continuous improvement and to treat nitpicks as optional. That position assumes reviewer enthusiasm is the scarce input. At a higher arrival rate, the scarce input is reviewer attention.

At that higher rate the same rule inverts quietly. A reviewer who approves anything broadly healthier now approves a great deal, quickly, and without the context to separate improvement from drift. Diginomica's report carries the operational lesson from early autonomous testing work: when people are pulled into a loop too often, they stop reading carefully and start accepting recommendations unexamined, so the design principle becomes to involve a person only where no other source of information can settle the question. That principle belongs in code review as much as in test triage.

Tooling has adapted unevenly. The National Cyber Security Centre's guidance recommends supportive tooling inside the development environment, naming linting, unit testing and specification checks among the things a team should provide, and its self-assessment puts automated style checking on the good side of the line. Each of those instruments produces findings that a person still has to interpret, and each is only as useful as the time the team reserves for that interpretation.

What the dark testing factory proposes, and what transfers

UiPath's argument, as Diginomica reports it, is that testing has to be scaled the way software is scaled, with headcount playing no part in the plan. The factory name comes from lights-out manufacturing, where production lines run with nobody on the floor. The substance is a lifecycle treated as one loop. Designing tests, generating data, provisioning environments, executing across platforms, analysing results and producing a release-readiness verdict all sit inside the same operating model.

The defining traits read as a specification. People sit above the loop, setting policy and resolving exceptions, and leave button-pressing to the system. The loop runs continuously inside delivery and waits for no nightly gate. A failure can trigger test repair, data reset, defect filing or escalation alongside the red mark. Scheduling follows risk drawn from change impact and production telemetry. Human decisions become reusable knowledge. Coverage deepens without proportional headcount. Diginomica sets the journey across five phases, from wiring the signals to governed autonomy, and the vendor is explicit that autonomy rises without ever reaching zero.

The detail worth copying is the cost architecture. Agents drive an application through its interface on the first pass, and deterministic Playwright scripts are authored as a byproduct of that exploration. Later runs replay the script without consuming model tokens. Playwright is the open-source browser framework that has largely displaced Selenium in web testing, and UiPath has built authoring into its Test Cloud product. Diginomica sets out the arithmetic that makes the pattern necessary: a single test case might run fifteen times a day, and across a year that multiplies into hundreds of thousands or even millions of executions, a volume that would make per-run model charges untenable. The factory also watches each component for variability. Where an agent returns the same output for the same input consistently, the system marks it as a candidate for deterministic replacement, which trims token spend without anyone making a judgement call.

What transfers to a team that will never buy the platform is the shape of that argument. Generation is expensive once; verification is expensive every time it runs. A quality process that pays full price on every repetition gets trimmed to what the budget tolerates, and the trimming lands first on the checks nobody watches. Holding quality steady under a faster release cycle therefore comes down to moving repeated judgement into something cheap and repeatable: a lint rule, a coverage gate, a scripted end-to-end path, a checklist item that fails a build.

Iouri Khramtsov, writing about managing quality while coding agents do more of the typing, sets out a layered arrangement he has seen work. Requirements get reviewed by a model for gaps and edge cases before any code exists. Unit tests are written from those requirements ahead of the implementation and pushed to near-total coverage. Manual testing survives for what a suite cannot see. End-to-end tests run on every change, in staging and after deployment. A separate pass hunts security problems and duplicated code. Human review and machine review both run on the pull request. Error tracking feeds production faults back into the queue. His claim is that the layers keep bugs steady while delivery speeds up, and that the cheaper checks get added precisely because agents make them cheap to write.

The layers matter because the defects do not announce themselves. NineTwoThree's account of what it calls the illusion of correctness makes the point sharply: generated code arrives tidy, consistently named and formatted in a modern style, and that tidiness gets mistaken for evidence that somebody checked the logic. Their reading is that fault counts can dip at first and climb months later, once the feature has shipped and the person who might have caught the problem has moved on.

None of this settles the question in the title for any particular team, because the answer is local. What the evidence supports is a way of asking it. Look at what happens to a change between the moment it is generated and the moment it reaches a user: how many independent checks it passes, how much of that checking is done by something that never gets tired, and how often a person signs off without building a model of what they signed. A team that can answer those three without guessing is in a position to speed up. A team that cannot is already guessing, and the release cadence will keep raising the stakes on the guess.

Sources

Source: Why is software quality declining - and can UiPath's Dark Testing Factory reverse the trend?, Diginomica

The usual questions

Why is code quality slipping despite AI tools?

Generation capacity has multiplied but verification capacity has not. Human attention is squeezed as delivery cycles compress, leading to a lag called acceleration whiplash.

What is the ghost code problem?

It is a situation where coding velocity rises while time-to-merge stagnates. This results in logic that runs without a clear lineage, human author, or long-term owner.

How can teams measure actual code health?

Teams should track lines rewritten within a fortnight, production incidents from the last release, and reviews approved within one minute of opening.

Filed so far