From a Single GPU to a Global AI Fleet: How LLM Inference Infrastructure Works

August 28, 2026

A practical guide to models, GPUs, serving, caching, batching, sharding, and intelligent routing

In one sentence: Serving a large language model is not just a matter of loading a file onto a GPU. It is a systems-engineering problem involving memory, bandwidth, scheduling, caching, networking, and orchestration. Artificial intelligence is often discussed as if it were purely a software breakthrough. At production scale, however, AI is equally an infrastructure challenge. The model may be a mathematical formula, but running that formula for thousands or millions of users requires an entire distributed system. This article explains that system from the ground up from the contents of a model file to the Kubernetes components that route requests across a global GPU fleet.

The big picture

A production LLM platform can be understood as a sequence of layers:

User request | v LLM-aware router | +--> Cache-aware scheduling +--> Load-aware scheduling +--> Session and prefix locality | v Model-serving layer | +--> Prefill workers ---- KV cache ----> Decode workers | v GPU fleet | +--> Single-GPU models +--> Sharded multi-GPU models +--> Multi-node Kubernetes workloads

The central challenge is simple to state: the model must remain available in memory while the system continuously moves data through extremely fast compute and memory pipelines.

1. The global scale of AI infrastructure

The growth of generative AI has triggered a large investment cycle in data centers, networking, power, cooling, and specialized accelerators. McKinsey estimates that global spending on data centers could approach $7 trillion by 2030.1 The exact investment figure varies by source, accounting method, and time period. The broader trend is clear: AI infrastructure is expanding from isolated GPU servers into large, geographically distributed fleets.

Infrastructure layer	Why it matters
GPUs and accelerators	Execute the matrix operations used by modern models
High-speed memory	Keeps model weights and temporary state close to the compute cores
Networking	Connects GPUs when a model or workload spans multiple devices
Model servers	Turn model files into APIs that applications can call
Schedulers and routers	Decide which worker should receive each request
Orchestration	Deploys, scales, monitors, and repairs the infrastructure

Key idea: At small scale, AI infrastructure looks like a GPU attached to a process. At large scale, it looks like a distributed operating system for models.

2. What is a model, really?

To understand the infrastructure, it helps to temporarily set aside the complexity of the word “AI.” At its core, a trained language model consists of a mathematical structure and a large collection of learned numbers.

The three essential parts

Part	Plain-language explanation
Architecture	The mathematical blueprint that describes how inputs are transformed into outputs
Weights	The learned numerical values stored inside that blueprint
Transformer	The highly parallel architecture used by most modern large language models
The architecture may be represented by a relatively small amount of code. The knowledge learned during training is primarily stored in the weights .
A useful mental model is to think of the weights as a very large data file. Running the model means loading that file into fast memory, converting the input text into tokens, and repeatedly applying the model’s mathematical operations to predict what comes next.

Why model size matters

The larger the model, the more memory it requires merely to remain available for inference.

Example model size	Approximate weight-file size*
Small model	2 GB
Medium model	16 GB
70-billion-parameter model	140 GB
Very large model	Several hundred GB or more
*Approximate sizes depend on the number of parameters, numerical precision, quantization, and file format.
This creates the first major infrastructure constraint: a model cannot run on a GPU unless the required weights fit into the available memory—or are divided across multiple GPUs.

3. Why GPUs rule LLM workloads

A traditional CPU is a general-purpose processor. It is designed to handle many different kinds of tasks, including operating-system work, branching logic, and sequential operations. LLM inference has a different shape. It performs enormous numbers of relatively simple mathematical operations, many of which can be executed at the same time.

CPU versus GPU

Characteristic	CPU	GPU
Design goal	General-purpose computing	Massively parallel computation
Strength	Flexible sequential logic	Large-scale matrix and vector operations
Typical core count	Relatively small	Thousands of simpler cores
LLM suitability	Useful for orchestration and preprocessing	Well suited to model computation
A CPU can run a language model, especially a small one, but it is usually inefficient for high-throughput production serving. GPUs are better suited because they can perform many operations in parallel.

The memory problem

Fast compute is useful only when data can reach the compute units quickly enough. If the GPU must repeatedly fetch model weights through a slower memory path, its cores spend time waiting rather than calculating. That is why GPUs use VRAM, or high-speed memory located directly on the accelerator. VRAM offers much higher bandwidth than ordinary system RAM, but it is also more expensive and limited in capacity.

System RAM GPU VRAM Large capacity Smaller capacity Lower bandwidth Much higher bandwidth Farther from GPU cores Located close to GPU cores Model weights must be available here for fast inference performance.

The three metrics that define an AI accelerator

When comparing GPUs, three measurements are especially important:

Compute: How quickly the accelerator can perform mathematical operations.
Capacity: How much model data and temporary state can fit in memory.
Bandwidth: How quickly the accelerator can read and write that memory. | GPU generation | Compute | VRAM capacity | Memory bandwidth | | --- | --- | --- | --- | | NVIDIA T4 | Source-dependent | 16 GB | Source-dependent | | NVIDIA A100 | Up to 312 TFLOPs, depending on precision | 80 GB | Approximately 2 TB/s | | NVIDIA H100 | Up to approximately 990 TFLOPs, depending on precision | 80 GB | Model-dependent | | NVIDIA H200 | Up to approximately 990 TFLOPs, depending on precision | 141 GB | Model-dependent | | NVIDIA B200 | Up to approximately 2,250 TFLOPs, depending on precision | 192 GB | Approximately 8 TB/s |

Important: Accelerator specifications vary by precision mode, product variant, and benchmark. Treat the table as an architectural comparison rather than a universal performance ranking.

4. From a model file to a production API

Loading a model into a GPU is not the same as serving it reliably to users. A production system needs a model server that can manage requests, schedule work, reuse memory, and expose a network interface. One widely used option is vLLM, an open-source inference and serving engine built for high-throughput language-model workloads. It can expose an OpenAI-compatible API, allowing existing applications to communicate with a self-hosted model with relatively few changes.2 A model server typically handles the following responsibilities:

Loading model weights into GPU memory.
Tokenizing incoming requests.
Scheduling multiple users efficiently.
Managing the key-value cache.
Streaming generated tokens back to clients.
Applying batching and memory-management strategies.
Reporting health, capacity, and performance metrics. The server itself can be large and expensive to start. Duplicating it for every request would waste resources, so production systems keep a controlled pool of long-running workers and route traffic intelligently among them.

5. The two phases of LLM inference

Language models do not process a response as one complete block. They generate tokens one at a time. A token may be a complete word, part of a word, punctuation, or another small unit of text. Inference is generally divided into two phases: prefill and decode.

Phase 1: Prefill

During prefill, the model reads the available prompt and processes it to create the internal state needed for generation. This phase performs a large amount of computation in parallel. The user usually experiences prefill as the pause between submitting a prompt and seeing the first generated token. This delay is commonly measured as time to first token, or TTFT.

Phase 2: Decode

During decode, the model generates the response token by token. The system repeats the prediction loop for every new token until the response is complete. Decode is often constrained more by memory bandwidth than by raw compute. The model must repeatedly access its weights and attention state while generating the next token. The resulting experience is commonly described using time per output token, or TPOT, and tokens per second.

Inference phase	What happens	Common bottleneck	User-visible metric
Prefill	The prompt is processed in a large initial pass	Compute	TTFT
Decode	The response is generated one token at a time	Memory bandwidth and cache capacity	TPOT / tokens per second

Why this distinction matters: A GPU configuration that is excellent for processing long prompts may not be the best configuration for streaming many responses. Prefill and decode place different demands on the hardware.

6. The optimizations that make serving practical

Without caching and batching, LLM serving would be far more expensive and much slower. Three techniques are especially important.

KV caching

The key-value cache, commonly called the KV cache, stores intermediate attention data produced while processing the prompt and earlier generated tokens. Instead of recomputing the same information for every new token, the server reuses the stored state. This reduces redundant work during the decode phase, but it consumes GPU memory.

Prefix caching

In a multi-turn conversation, much of the prompt may remain unchanged from one request to the next. Prefix caching reuses the cached computation associated with that repeated prefix. This can reduce repeated prefill work, especially when many requests share the same system instructions, document context, or conversation history. vLLM documents prefix caching as one of its serving optimizations.3

Batching

A GPU can often serve multiple users more efficiently than it can serve each user independently. Batching combines work from several active requests so that the model weights are reused across them. Modern serving systems commonly use continuous batching, which allows new requests to join the workload while other requests are still generating output. This keeps the GPU busier than waiting for an entire batch to finish before starting another one.4

Without batching: User A -> load/read model -> generate token User B -> load/read model -> generate token User C -> load/read model -> generate token With batching: User A ----+ User B ----+--> shared model execution --> tokens for A, B, C User C ----+

The memory ceiling

Batching is not free. Each active request may require its own KV-cache space. The model weights occupy a fixed portion of VRAM, and the remaining memory determines how many requests can be active at once. This leads to a counterintuitive production behavior: a GPU can have idle compute cores and still be at capacity because its memory is full.

7. Sharding models across GPUs

What happens when a model is larger than the memory of one GPU? The model must be divided across multiple devices. This is known as sharding or distributed model execution.

Tensor-style splitting

One approach divides operations within a layer across several GPUs. The devices must exchange intermediate results frequently, so this strategy requires very fast interconnects such as NVLink. It is usually most practical within a single server or tightly coupled GPU system.

Pipeline-style splitting

Another approach assigns different layers to different GPUs or machines. Each request passes through the layers in sequence, like an assembly line. This reduces the amount of communication required between devices, making it more suitable for some multi-node deployments. The trade-off is that the request must travel through multiple stages, and pipeline scheduling becomes important.

Sharding approach	What is divided?	Communication need	Typical fit
Tensor parallelism	Work inside a layer	Very high	GPUs in one tightly connected server
Pipeline parallelism	Groups of layers	Lower between stages	Multi-stage or multi-node deployments

Rule of thumb: The more frequently GPUs must exchange intermediate data, the more important the network between them becomes.

8. Why traditional load balancers are not enough

Round-robin load balancing works well when requests are mostly independent and servers are interchangeable. LLM requests do not behave that way.

Problem 1: Traditional routing is not cache-aware

Suppose a user sends a second message in an ongoing conversation. A conventional load balancer may send it to a different server from the one that handled the first message. The new server does not have the relevant cached state. It must process the repeated prompt again, increasing latency and compute cost.

Problem 2: Requests are not equally expensive

A short greeting and a request to summarize a long document may look identical to a basic HTTP load balancer. In reality, their prompt lengths, generation lengths, memory requirements, and scheduling impact can be very different. A large prompt routed to an already saturated worker can increase latency for every user sharing that worker.

Traditional assumption	Reality in LLM serving
Servers are interchangeable	Cache location affects performance
Requests have similar cost	Prompt and output lengths vary widely
CPU or connection count indicates load	VRAM use, queue depth, and token workload matter
Any healthy worker is a good destination	The best worker depends on cache and capacity

9. LLM-aware routing with Kubernetes

The llm-d project is an open-source initiative focused on distributed inference serving on Kubernetes. It adds LLM-aware routing and scheduling concepts to the orchestration environment many organizations already use for production workloads.5

Cache-aware routing

An LLM-aware router can track where relevant cached state is available and prefer that worker for subsequent requests. Reusing the cache can reduce repeated computation and improve response latency.

Load-aware routing

Instead of looking only at connection counts, an inference-aware router can consider signals such as:

Available VRAM.
KV-cache usage.
Queue length.
Number of active sequences.
Prompt length.
Expected generation workload. This produces a more realistic picture of worker capacity.

Disaggregated inference

Because prefill and decode have different bottlenecks, a deployment can place them in separate pools:

In this model, one pool processes prompts while another pool generates tokens. The KV cache is transferred between the stages over a fast interconnect or network path. This approach can improve utilization when the workload contains a strong mix of prompt processing and response generation. Actual gains depend on model architecture, request distribution, hardware, network speed, and implementation details.

Coordinated scaling and recovery

A sharded model is not a collection of independent pods from the application’s perspective. Its workers must be scheduled, scaled, and recovered as a coordinated group. Kubernetes mechanisms such as the LeaderWorkerSet API are designed to help manage groups of tightly related workers as a single logical workload.6

10. A practical architecture checklist

When designing an LLM-serving platform, ask the following questions in order:

Question	Why it matters
Does the model fit on one GPU?	Determines whether sharding is necessary
How much VRAM remains after loading the weights?	Limits batch size and KV-cache capacity
Is the workload prompt-heavy or generation-heavy?	Helps choose prefill and decode hardware
Are conversations long or repetitive?	Determines the value of prefix caching
How variable are request sizes?	Influences scheduling and admission control
Does the model span multiple GPUs or nodes?	Determines interconnect and orchestration requirements
Can the router observe cache and memory state?	Enables inference-aware routing
Can workers scale and recover together?	Protects availability for sharded deployments

Conclusion: infrastructure professionals are already close to the answer

Large-scale AI infrastructure may sound like an entirely new discipline, but much of it builds on familiar systems concepts. Linux, containers, Kubernetes, observability, networking, capacity planning, and failure recovery remain essential. The new layer is an understanding of GPU memory, model serving, token-generation behavior, KV caching, and inference-aware scheduling.

The opportunity: Infrastructure engineers do not need to become model researchers to contribute to the AI era. By adding GPU and inference knowledge to existing platform-engineering skills, they can solve some of the most important problems involved in making AI reliable, fast, and economical. The path from one model on one GPU to a global AI fleet is therefore not a single leap. It is a sequence of engineering decisions: fit the model into memory, move data efficiently, reuse computation, batch requests, shard when necessary, and route every request according to the real state of the system.

Quick glossary

Term	Meaning
Inference	Running a trained model to produce an output
Token	A small unit of text processed by a language model
VRAM	High-speed memory attached to a GPU
Prefill	Processing the input prompt before generation begins
Decode	Generating the output one token at a time
TTFT	Time to first token
TPOT	Time per output token
KV cache	Stored attention state reused during generation
Prefix caching	Reuse of computation for repeated prompt prefixes
Batching	Processing multiple requests together
Sharding	Splitting a model or its computation across devices
LLM-aware routing	Routing based on cache, memory, queue, and token workload

References

Editorial note: The infrastructure figures in the original source—including GPU throughput, token-generation limits, and claimed percentage improvements—depend on hardware, precision, model, workload, and benchmark methodology. They have therefore been presented here as approximate or contextual rather than universal guarantees.