Serving large language models at scale puts infrastructure teams in a blunt spot with the KV cache. Either you provision oversized GPU instances just to fit a swelling cache, or you live with slow time-to-first-token because the same prompts keep getting recomputed. AWS now describes a tiered design on SageMaker HyperPod, built with the Curvine distributed cache, that sidesteps that choice by sharing key-value caches across GPU replicas instead of re-prefilling them. The full explanation is on the AWS Machine Learning blog.
The mechanics are straightforward. As vLLM generates text, it keeps the attention key and value vectors for each token it has already handled in a KV cache, which avoids recomputation on later steps. Prefix caching builds on that by letting different requests with matching opening tokens, for example a shared system prompt, reuse the same cached state. On economical hardware such as the ml.g6e.4xlarge with 48 GB per GPU, only a sliver of memory remains for prefix caching once weights and runtime overhead are accounted for, and that sliver shrinks further as models grow or concurrency rises, as the AWS Machine Learning blog notes.
The trouble surfaces in three ways. Hit rates fall on long prompts. The same system prompt gets recomputed on every call. And when vLLM scales out, each replica holds its own separate cache, so sending a request to another replica is basically a cold start. For shops serving a wide set of open foundation models like Qwen, Llama, and DeepSeek across separate per-team endpoints, RAG pipelines, or multi-turn chat, that pattern means higher infrastructure bills and a worse experience for users, per the same post.
The remedy is a three-level cache that stretches past a single pod. L0 sits in GPU high-bandwidth memory and holds vLLM’s native paged-attention layer, the hottest KV blocks at the lowest latency; its room is just whatever GPU memory is left after the model weights load. With a 48 GB GPU, a 7B model in bf16 eats about 14 GB for weights, freeing more than 30 GB for KV blocks, so L0 rarely strains. A 32B model needs roughly 64 GB just for weights and won’t fit on one 48 GB GPU; once sharded, little memory is left for the cache, which then fills and drops blocks under load. These specifics come from the AWS walkthrough.
L1 is the CPU-memory safety net. When GPU blocks get evicted, LMCache lives inside each inference pod and intercepts them in the host’s DRAM so the work isn’t discarded. The HyperPod Inference Operator switches it on automatically whenever the InferenceEndpointConfig custom resource sets enableL1Cache: true, and AWS suggests beginning with InstanceMemoryAllocationPercentage at 20 percent to size the buffer. The outcome is a fast, pod-local cache managed for you, as the AWS blog describes.
L2 is what delivers cross-replica reuse, and Curvine fills that slot. Curvine is a lightweight distributed cache filesystem that gathers the local NVMe drives on G6e and P5 instances into one shared namespace. A FUSE client presents that pool as a ReadWriteMany volume mounted into every inference pod. Through its fs:// connector, LMCache treats the shared pool as an ordinary local folder, and because all pods see the same namespace, a KV block one replica writes is instantly available to the others. Curvine’s design puts a Primary node, called the Master in its documentation, in charge of metadata and journaling that persists on Amazon EBS for durability, while Worker components run on each GPU node and store data on node NVMe, typically at /opt/dlami/nvme/curvine-data. The setup is laid out in the AWS post.
Curvine’s internal flow is deliberately easy to run. Clients send metadata requests to the Masters and send their actual data reads and writes to the Workers. The Masters keep Workers in sync via heartbeats and decide block placement for load balancing and availability. Workers handle local tiers and shift data up or down in the hierarchy based on how hot it is. On a miss, or when a policy forces persistence, Curvine pulls from or writes to the underlying file system, so durability stays on the backing store while Curvine speeds up access, as the AWS post describes.
A cache only helps if requests reach the replica that already holds the right blocks. The HyperPod Inference Operator ships a built-in router with three strategies. It keeps either a prefix tree or a per-worker cache map to route each request to the replica most likely to score a hit. This is invisible to clients, with no code changes required. SageMaker HyperPod itself is AWS’s resilient cluster service for large-scale training and inference across thousands of accelerators such as Trainium and NVIDIA H100 GPUs, and its managed tiered KV caching plus intelligent routing are part of the deployment surface, according to the HyperPod documentation and the model deployment guide.
In a test deployment, the full stack hit up to a 100 percent cross-pod cache hit rate, improved TTFT by as much as 2.7x, and showed a cross-node L2 read latency around 56 ms for a prompt of roughly 1,900 tokens, AWS reports in its benchmarking section. The cost angle is the real draw: workloads that once needed P5 instances can instead run on cheaper G6e hardware, lowering per-endpoint cost, though the actual saving varies with model size and traffic shape.
HyperPod’s built-in path already provides a two-tier setup: a CPU-based L1 plus an L2 that uses Redis for node-spanning cache sharing, while intelligent routing sends each request to the instance most likely to already hold the needed key-value pairs, according to the model deployment documentation. The Curvine approach substitutes a distributed NVMe pool for Redis, which matters when you already have fast local NVMe on G6e or P5 boxes and want cross-node reuse without standing up a separate Redis tier.
The wins scale with how much prompts overlap. When leading tokens are shared by more than about 40 percent, as with a common system prompt or a shared RAG context, avoiding the re-prefill sharply cuts TTFT. The request path is clean: a request hits the router, gets sent to the replica with the best prefix match, that replica checks GPU blocks, then CPU, then the shared NVMe pool, and only on a total miss does it re-prefill from scratch. This flow is detailed in the AWS write-up.
This design isn’t a blanket speedup. AWS ties the gains to that moderate-to-high overlap, again roughly above 40 percent shared opening tokens. For traffic with little shared prefix, think distinct single-turn prompts, the L2 pool rarely hits and the routing plus FUSE overhead may add nothing. So the published 2.7x TTFT gain is a ceiling for overlap-heavy workloads, not a default for every endpoint, as the AWS benchmark notes.
Getting it running has clear prerequisites. SageMaker HyperPod Tiered Storage is a cluster-level feature that provisions a node-local cache tier; after it’s on, HyperPod pushes the ai-toolkit DaemonSet to each GPU node, sets aside a configurable slice of host memory for the L1 offload, and surfaces the node’s local NVMe at /opt/dlami/nvme so Curvine Workers can aggregate it. On the operator side, AWS expects CLI v2 with sagemaker:UpdateCluster and eks:CreateAddon rights, a kubectl context for the cluster, and Helm v3. The EBS CSI driver’s IAM role must hold sagemaker:AttachClusterNodeVolume, sagemaker:DetachClusterNodeVolume, and eks:Describe*, otherwise the Curvine metadata node can’t attach its EBS volume. InstanceMemoryAllocationPercentage accepts anything from 20 to 100; AWS advises starting at 20 and raising it as throughput and hit rate dictate. These steps are catalogued in the AWS guide.
AWS splits the build into five stages using a worked example. Stage 1 turns on Tiered Storage through an update-cluster call that sets Mode=Enable and InstanceMemoryAllocationPercentage=20; because that call rejects a tiered-storage flag on its own with a ValidationException, you also feed back the current NodeRecovery value pulled from describe-cluster, which changes nothing else. Stage 2 adds the Inference Operator and its dependencies, either via the console’s Quick Install or as EKS add-ons covering the S3 and FSx CSI drivers, Metrics Server, Cert Manager, and the inference add-on. Stage 3 deploys Curvine, Stage 4 patches the Operator for the filesystem-backed L2, and the final stage checks the end-to-end path. The procedure is reproduced in the AWS walkthrough.
One caveat remains today. The InferenceEndpointConfig CRD’s l2CacheBackend field natively takes only redis or tieredstorage. To aim L2 at a Curvine FUSE mount instead, AWS edits the LMCACHE_REMOTE_URL environment variable inside the vLLM container spec to fs://localhost:0/mnt/curvine/l2cache/. You set the desired cache topology in the CRD using enableL1Cache, enableL2Cache, l2CacheBackend, and routingStrategy, and the Operator then produces the matching environment variables, volume mounts, and routing rules. This wiring is laid out in the AWS guide.
For platform and infrastructure leaders, the budget case is as important as the technical one. AWS points out that serving a catalog of open models through separate endpoints usually means each replica carries its own cold cache, so teams pay for either bigger GPUs or slower responses. By pooling KV state on shared NVMe and steering requests to warm replicas, the Curvine-backed design lets teams consolidate on G6e-class hardware instead of P5, which is where the documented per-endpoint cost reduction comes from. The operational trade is that you now run a Curvine cluster alongside the inference plane and own its metadata durability on EBS.
For teams already on AWS’s Kubernetes stack, the surrounding tooling is familiar. Running interactive development environments on Amazon EKS with SageMaker AI follows the same EKS-oriented operational model that this cache architecture builds on, as shown in zBrandco’s walkthrough of interactive IDEs on EKS with SageMaker AI. And the tiered-cache idea is not unique to LLM serving: Cloudflare’s Smart Tiered Cache uses a similar regional-hint concept for content delivery, covered in zBrandco’s Cloudflare tiered cache guide.
The bottom line is concrete. If you serve multiple models or share system prompts across endpoints, the Curvine-backed L2 tier turns isolated GPU caches into one shared pool, and the published benchmarks suggest meaningful TTFT gains without moving to the most expensive instance class. As with any preview-era capability, the savings are workload-dependent, so the cited 2.7x figure should be read as a measured upper bound rather than a guaranteed default, per the AWS benchmark methodology.
