Over the past year the question platform teams hear about generative features has moved from how to ship one to who owns it once it reaches production. The answer forming inside UK engineering organisations is a shared layer between applications and models, holding the identity, budget, version history and audit trail that product teams used to improvise project by project. Building that layer costs a component to run and a contract to publish; operating it saves the weeks that otherwise pass between a prompt change and the invoice that shows what the change did.
The assumptions that stop holding
In a conventional service a request is a bounded unit of work. The same code path costs roughly the same each time, replicas are interchangeable, and extra capacity is a purchasing decision with a known price. Put a model inside the request and those three assumptions fail. Compute rises with the length of the prompt, so two calls through one function can differ by orders of magnitude. Two pods loaded with the same weights stop being twins, because one may already hold a conversation's attention state and answer much faster. Accelerator capacity is usually reserved ahead, so the machine a spike needs may not be available to buy.
Two consequences follow. A handful of controls only work when every request meets them at the same point, and they belong together: tenancy and identity, token metering, guardrails on input and output, and routing across providers. Leave those to individual services and the organisation ends up with spend it cannot attribute, no kill switch and failover nobody has tested. The gateway is the one place where the set can be enforced once, and the reason to install it first is arithmetic: once four teams hold provider keys, nothing upstream can cap what they spend.
model_list:
- model_name: fast-tier
litellm_params:
model: openai/qwen2.5-7b-instruct
api_base: http://vllm-small.inference.svc.cluster.local:8000/v1
api_key: os.environ/INTERNAL_KEY
- model_name: reasoning-tier
litellm_params:
model: anthropic/claude-sonnet-4-6
api_key: os.environ/ANTHROPIC_KEY
litellm_settings:
callbacks: ["otel"]
cache: true
cache_params:
type: redis
host: redis.platform.svc.cluster.local
general_settings:
master_key: os.environ/GATEWAY_MASTER_KEY
max_budget: 20000
budget_duration: 30d
That file holds routing, caching, tracing hooks and a monthly ceiling. A stronger open model arriving next quarter becomes a mapping edit, and application teams notice only through a lower bill and faster answers.
The second consequence concerns what counts as a deployment. Model weights carry a version, a quantisation setting and an engine configuration. Prompt templates carry the tool schemas a model may call. The embedding model carries its chunking rules and the index built from them. The guardrail policy set and the evaluation data that defines a pass travel with the same discipline. An embedding swap invalidates every stored vector at once, because each model defines its own vector space, so a corpus rebuild belongs in a runbook with a measured duration long before anyone schedules the upgrade.
A prompt is a change to behaviour, and a change without a fixed evaluation set cannot be compared with what came before. The engineering practices published on martinfowler.com after a short proof-of-concept engagement remain the most practical starting set: example-based tests that assert on a structured field such as intent, property tests scored by an evaluator model, and adversarial cases collected from every failure the team can imagine. The same article is honest about the limit, since a test can show that a bug exists and can never show that none remain.
Production results are worth knowing. TechGig reported that a retail inventory system brought its production hallucination rate down from 15 per cent to 1.5 per cent with the foundation model unchanged, through classifying failures before any retry, adding an intent gate allowed to answer "unclassified", and keeping prompts in a registry that made each instruction change reversible. The longer account on InfoQ describes the platform mechanics: schemas enforced at the platform boundary, responses validated against them and re-prompted with the validation error attached, and failures sorted into schema violations, grounding failures and infrastructure errors so that each retry carries new evidence.
Open Source For You's account of the same shift, published in October, describes a logistics team in Pune whose model spend climbed from Rs 42,000 in February to Rs 1.1 million by the middle of April while nobody could name the service responsible. The lesson is that most of the existing estate survives the transition: the cluster still schedules pods, the pipeline still ships them, and a small number of assumptions break underneath.
What the platform has to make visible and enforceable
Three audiences read one event stream with different questions. The on-call engineer needs to know whether to add capacity, the product owner needs to know whether quality moved, and finance needs to know which team spent the money. A dashboard built for all three answers none of them well, so the platform keeps one pipeline and publishes three views over it.
The signals that predict user pain sit outside a standard service dashboard. GPU utilisation reports that a kernel ran and says nothing about useful work, which makes it the least trustworthy figure on the board. Queue wait is the early warning, because first-token time rises steeply once requests wait for cache memory. An average across the whole request can look healthy while a stream stalls in the middle, since the response eventually completes. Goodput, the share of requests served inside the objective, tells a team when to add capacity, and tokens per second per accelerator read beside utilisation tells it whether the hardware earns its place.
Tracing follows the same principle. One question fans out into retrieval, several model calls and a few tool invocations, and the useful artefact is a single trace with each of those as a span nested inside the request that caused it. Open Source For You reports that OpenTelemetry's generative AI conventions now provide shared field names for the model, the tokens consumed, sampling temperature, why generation stopped and any tool calls, so a hosted call and a locally served call land in the same trace store. The conventions remain in development, so pin the convention release alongside the instrumentation packages and read one stored trace before relying on a documentation example. Spans leave prompt and completion text unrecorded unless a team opts in. What is kept is sampled and held in a separate store with its own retention and access policy.
Quality needs its own evidence. Evaluators can grade a sample of live traffic, while implicit signals come free: a retry, an edit or an escalation to a person each marks an answer that did not land. A judge model adds a second bill and its own noise, so a sample is the sensible default.
Cost needs its own vocabulary. A price per token belongs in accounts. It drops when traffic moves to a cheaper model, including when that model needs several attempts before an answer passes. The number engineering should defend is cost per successful task, which carries retries, handoffs to people and abandoned sessions inside it. Every request should carry its team, use case, model and environment from the first production call. A label added later splits the history it was meant to explain. LaunchDarkly's pricing guidance sets out the deployment choice behind the number and the practices that keep it visible.
- Hosted API: charged per token with no infrastructure to run, suited to prototypes and low or medium volume.
- Self-hosted on cloud GPUs: charged by the hour whether traffic arrives or not, suited to steady high volume or data that cannot leave the estate.
Security starts with a drawing of trust boundaries, because the intuitive drawing is wrong. A model gets commands and content through the same input, and cannot reliably separate the two. The direct form of injection is familiar. The indirect form carries the greater risk: someone hides a command in a support ticket or a document, retrieval brings it into the prompt, and the model treats it as direction. Filters will miss some attempts, so the design should limit what a successful one can achieve. Each tool credential should carry the minimum rights and act with the end user's authority. A schema check should run before any downstream system consumes model output. Human sign-off belongs on payments, outbound messages and deletions. Outbound traffic from a tool sandbox should pass an allowlist. A session needs hard limits on how many loops and tool calls it can make and how many tokens it can emit.
Model files are executable artefacts, and the supply chain around them deserves the same discipline as container images. Python pickle files can execute code when loaded, so the only accepted path should take safetensors and reject pickle unless a documented exception exists. Provenance should come from a signature. Scanning should cover the artefact and its dependency graph, and the result should be recorded as an attestation. An AI bill of materials should record where the model came from, what data trained it and how it scored. Approved weights belong in an internal registry, and production networks should have no route to public hubs.
Written standards do not stop a pod from starting. Admission policy does. A cluster policy can require model artefacts to come from the internal registry, deny inference pods without a cost-attribution label, demand a resource claim in place of a bare GPU count, and require a signature attestation on any image that loads weights. Those rules turn a platform standard into a runtime fact.
Delivery splits into three streams at different tempos, all pointing at one declared state. Application code keeps its normal cadence, prompts and tool schemas move through a registry, and model versions pass the evaluation gate. A release should stop when evaluation scores, forecast cost per task or canary quality fall below the agreed line. Shadow traffic runs a second model against the same live inputs and keeps its answers away from users. Canary by request class teaches more than a slice of all traffic, since sending one low-risk query type through the new version exercises the path that matters. A rollback path needs the prior model kept warm and the prompt version it was tested with kept beside it.
High availability changes shape when accelerators are scarce. Keeping a spare accelerator idle costs money every hour, so failover across providers at the gateway often buys more resilience for less, as long as someone tests it. A failover path nobody has run is a guess. The gateway can also hold a pre-agreed ladder of degraded modes, so an outage becomes a slower service with fewer features. The last rung deserves attention, because a product with no fallback outside the model suffers a complete outage when inference fails.
Regulated deployments need one more control: an immutable record that links the request, prompt version, model version, retrieved document identifiers, tool calls, output and any approver. That record is what makes an audit answerable without asking an engineer to reconstruct a session from logs.
Sequence matters. Put the gateway in first, because it gives identity, metering, routing and a kill switch in one component. Instrument to the generative AI conventions next with content capture off. Build the evaluation suite third, since routing, caching and model changes all depend on it. Optimise cost last, because caching and routing without evaluations is optimisation without a safety net.
The platform layer earns its keep by letting a UK team ship a second and a third model feature without rebuilding the first one's controls, and by turning a generative prototype into a service the organisation can price, audit and defend.
Sources
- Platform Engineering For LLM Applications - Open Source For You
- Platform engineering crucial for production LLM reliability and cost control - TechGig
- Engineering Practices for LLM Application Development - martinfowler.com
- The Platform Engineering Playbook for Production LLMs - InfoQ
- LLM pricing comparison: Tutorial and best practices - LaunchDarkly
Source: A Peek At Platform Engineering For LLM Applications, Open Source For You


