AI-Driven Code: How to Ensure System Reliability

"If the assistant wrote it and the tests are green, what is a reviewer actually checking?"

The reviewer is checking everything the model could not know. Which failures this service is allowed to have. Which dependencies are approved. What classification of data the endpoint touches. What the code path does when the database stops answering. None of that is in the prompt unless someone put it there, and none of it is in the test suite unless someone wrote a test for it. Generated code can be readable, syntactically correct and still wrong for the system it lands in.

That gap has widened because code volume has grown faster than the review capacity around it. The report behind this page, Vocal's account of building reliable systems in an AI-driven world, argues that the next period of software engineering will be judged by how well systems are built, tested, secured and improved, and it describes the shift as one in which "speed and reliability must evolve together". The failure modes it names are specific: reviews that cannot absorb the volume, dependencies nobody vetted, test suites that confirm the happy path, and technical debt accumulating faster than a team can clear it. It also says plainly that engineers keep the judgement call and need better tools and clearer information before they commit.

Two things follow, and they are the two halves of this page. Before the merge, you need a gate that a named person closes and a record showing they closed it. After the merge, you need signals that read real traffic, because the failures that matter now pass the test suite and show up later in latency, error rates and cost. The first half is a configuration problem. The second half is an observability problem. Neither is solved by buying a better assistant.

What the reviewer signs, and how the repository remembers it

Start with the control everything else hangs from: a human approves the change before it reaches production, and the approval is attributable to a person. The Home Office's engineering standard for AI use, SEGAS-00020, makes this a hard requirement, alongside traceability through commits, pull requests and reviews, testing to existing standards, approved tools only, and the same security expectations for AI-assisted code as for anything else. The framing of responsibility is the part worth reading twice. AI systems count as tools, and teams keep full accountability for the output. The standard also tells teams to plan for the worst, on the assumption that generated output may be incorrect in ways that are not obvious where it was introduced.

A standard on a wiki changes nothing. Put the marker in the commit and make the pipeline read it. A trailer is enough:

feat(invoices): validate currency before totals

AI-Assisted: copilot
Reviewed-By: @j.okafor

Then a check that refuses to merge AI-assisted work with no named reviewer. Keep it dull on purpose: it fails closed, it needs no external service, and it leaves an audit trail in the history.

#!/usr/bin/env bash
set -euo pipefail

base="${CI_MERGE_REQUEST_TARGET_BRANCH_SHA}"
range="$base..HEAD"

if git log "$range" --format=%B | grep -q '^AI-Assisted:'; then
  reviewers=$(git log "$range" --format=%B | grep -c '^Reviewed-By:' || true)
  if [ "$reviewers" -lt 1 ]; then
    echo "AI-assisted work needs a named reviewer in the commit trailer."
    exit 1
  fi
fi

Wire it into the merge request pipeline as a blocking check:

ai-review-gate:
  stage: verify
  script:
    - ./scripts/require-ai-review.sh
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

The prompt deserves the same treatment. It is a configuration file for a non-deterministic component, and it should be written, reviewed and versioned like one. Matt Tanner's guide for Speedscale makes the case that the prompt is the largest single factor in what comes back, and that a vague request leaves the model to invent security, performance and error handling for itself. A prompt that names its constraints gives the reviewer something to check against:

Task: add GET /api/invoices/{id} to the existing Go service.

Pattern:
- Follow internal/http/handlers.go and wrap with LoggingMiddleware.

Imports:
- net/http, encoding/json, time, github.com/rs/zerolog/log.
- If another package is needed, give the exact import path and confirm the
  function exists in the version pinned in go.mod.

Behaviour:
- 400 for a malformed id, 404 for a missing invoice, 500 with a generic body
  for anything else.
- 5s timeout on the database call, context passed down from the request.

Logging:
- Structured fields only. No formatted string containing user input.

Tests:
- Table-driven cases for 400 and 404, plus a test asserting the timeout.

That prompt is longer than most tickets and it earns its length. Every line is a review question nobody has to reconstruct from the diff.

When the diff arrives, read it for the patterns that compile cleanly and fail in production. These are worth checking by name:

• Hallucinated symbols. The function name looks right for the library and does not exist in the pinned version. Check the import and the call; the prose around it will read convincingly.
• Unsafe defaults. Certificate verification switched off, a debug endpoint left reachable, a query built by string concatenation, a credential in a placeholder. The model reproduces what it learned, and some of what it learned is insecure.
• Happy-path logic. The input is accepted, the response is built, the error path is a comment. Ask what happens on a timeout, a malformed payload and a duplicate submission.
• Unbounded work. A loop over a collection, an API call inside a loop, a buffer that grows with input. Speedscale's list of common reliability failures names this class directly, because the code is functionally correct on small data.
• Stale dependency versions. The model was trained before the current release and will suggest a pattern that has since been replaced. Check the lockfile against the suggestion.
• Tests that restate the implementation. Nobl9's guide to the risks of AI-generated code describes generated tests that mirror the logic under test, so a wrong implementation and its test agree with each other. A high coverage number is not evidence of anything when both sides came from the same assumption.
• Missing authorisation. The handler reaches the data store without checking who is asking. This one rarely appears in a unit test and rarely survives staging.

The supply chain needs its own gate. The NCSC's guidelines for secure AI system development treat security as a core requirement across design, development, deployment, and operation and maintenance, and they name supply chain security, documentation and technical debt management as part of development. Generated dependencies are exactly that problem: a package name that sounds plausible, resolved from a registry, pinned in a lockfile, never read.

Attribution is the harder half. Cycode's 2026 State of Product Security Report found that 97% of organisations are using or piloting AI coding assistants and that 81% lack visibility into how and where AI is used across the software development lifecycle. Without that visibility, a security team cannot tell which code came from a model, cannot apply a scanning policy to it, and cannot trace a vulnerability back to the tool that produced it. If nobody can answer which assistant generated a file, the review gate is the only control left standing.

The failures that only appear on your traffic

A green pipeline says the logic holds for the inputs the tests provided. It says nothing about the distribution of latency, the rate at which errors consume an error budget, or the behaviour of a dependency under load. Nobl9's guide to the risks of AI-generated code describes the pattern: the code looks correct and passes basic tests, then shows up in production as rising P95 latency, higher error rates, unnecessary retries and increased cloud costs. The guide lists the design gaps that produce it, including missing retries, timeouts, rate limits and circuit breakers.

Set a reliability target before the review and the argument changes shape. Nobl9 gives the worked example, and the targets do the reviewing:

• Availability at 99.9%. The design needs fallback paths and isolation boundaries.
• Latency at 100 milliseconds. Long chains of synchronous calls are out.
• A small error budget. Retry policies, backpressure and timeout settings become mandatory.
• A P95 latency ceiling. The review asks what the slow path does at peak load.
• An error-rate ceiling for each endpoint. A global average can hide one route failing for one group of users.

Each line is a question with a number or a limit attached, and a reviewer can point at the design and ask where it is answered.

The drift is gradual. A minor latency increase persists for several releases, each one passing its tests, and the error budget drains before anyone connects the two. Nobl9 calls this reliability drift and recommends SLO checks in CI/CD, canary analysis and composite SLO monitoring for multi-service changes. The release gate reads correctness. The production signal reads efficiency and stability under load, and those are the properties generated code most often changes without saying so.

The operational answer has a maturity order. Smicolon's guide to AI code reliability sets out four stages, and each one grants the system a little more authority. An observability foundation comes first: logs, metrics, traces, service ownership, deploy markers, incident records and dependency maps have to be consistent enough that a responder can trace a customer-facing symptom back to the services and changes behind it. Read-only AI assistance comes second: the system watches for anomalies, summarises incidents, correlates deploys, retrieves past incidents and drafts hypotheses, while holding no production write access, and success is measured in time-to-context, alert precision and hypothesis accuracy. Assisted actions come third: the system prepares selected actions for human approval, such as drafting a rollback or isolating an unhealthy node, with role-based access control limiting what it can touch and audit logs recording every step. Governed self-healing comes last: autonomous remediation is authorised only for repetitive, well-understood incidents with tested responses, narrow scopes, explicit stop conditions and strict escalation rules.

The reason to keep the stages in order is visible in the comparison between a manual response and an AI-assisted one.

Manual response. Detection waits for a fixed alert to cross a static threshold. Recent changes are found by searching deploy tools and asking the team. Impact is understood by inspecting several dashboards by hand. A hypothesis is formed by correlating evidence manually. The team decides and executes the fix.

AI-assisted response. Detection uses anomaly detection across related signals. An incident summary lists relevant deploys and config changes. The system groups affected services, journeys and dependencies. The AI drafts ranked hypotheses with supporting evidence. A human approves a suggested action or follows a documented rollback path.

Smicolon's guide to AI code reliability cites Google SRE's reported results: AI-generated incident hypotheses reduced mean time to mitigate by 10%, and investigation dashboards surfaced 195% more findings and cut mean time to mitigate by roughly 44% for supported incidents. The engineer still checks the hypothesis. The AI assembles the case, and the engineer applies judgement.

Any system allowed to act in production needs controls that a reviewer can inspect:

• Role-based access control tied to explicit roles and scopes.
• Mandatory human sign-off for higher-risk actions.
• Audit trails covering evidence, recommendations, approvals and execution logs.
• Automatic stop conditions that halt automation when uncertainty rises.
• Tested rollback paths for every automated change.

That list is the boundary between assistance and an outage with a fancier dashboard.

The model itself belongs in the same argument when the product ships one. Monte Carlo's AI reliability guide treats model drift as a production reliability problem: data distributions shift, accuracy degrades, and monitoring has to compare incoming data with training data and trigger retraining when drift crosses a threshold. The guide also recommends simulation testing, stress testing, user acceptance testing and a pilot rollout before scaling up, which is the same gate logic applied to a model. The code review gate and the production signal are two ends of one process. A team that closes the first and ignores the second will ship generated code that passes every check it wrote for itself.

Sources

• SEGAS-00020 Use AI
• A Developer's Guide to Improving AI Code Reliability
• Guidelines for secure AI system development
• AI code security: Complete guide
• A Guide to the Risks of AI Generated Code
• How AI Helps With Code Reliability When Your Product Changes Faster Than Ops Can Keep Up
• The AI Reliability Guide: How To Build Reliable AI Models That Don't Fail

Source: The Future of Software Engineering: Building Reliable Systems in an AI-Driven World, Vocal