SMART WORLDJOURNALSubmit Manuscript
← All technology news

Artificial Intelligence

Amazon’s Distributed KV Caching Skill Slashes Cost and Latency for Self-Hosted Large Language Models

Amazon Web Services introduces a distributed key-value caching skill using ElastiCache to optimize GPU workloads for large language models, reducing inference costs by up to 40% and initial token latency by up to 80% across complex, long-context AI applications.

NEWS

As enterprises self-host large language models (LLMs) on GPUs for better control over expenses, accuracy, and response times, redundant computation remains a key bottleneck. Each GPU server keeps isolated caches of key-value (KV) attention states, which cannot be shared effectively. This results in repeated, costly recomputations—known as prefill—when resuming long conversations with large contexts, such as legal AI assistants handling extensive case histories.

To address this, AWS unveiled a distributed KV caching skill integrated with Amazon ElastiCache. By extending beyond local GPU caches to a shared cache layer accessible across the server fleet, any GPU can reuse cached KV states for days regardless of session interruptions or server changes. This approach cuts inference costs by up to 40% and decreases the time to first token (TTFT) by as much as 80%, greatly enhancing LLM efficiency on workloads with large token contexts.

The KV caching skill is part of the Agent Toolkit for AWS and is compatible with coding agents like Kiro and Claude Code. It streamlines setup through an intelligent, prompt-driven process that diagnoses workload suitability, estimates cost savings using the KVTrace tool, configures ElastiCache clusters, and assists in troubleshooting. By reading server configurations and usage patterns, the agent calculates if distributed caching provides net benefits before infrastructure is deployed.

A key capability is KVTrace, which analyzes real traffic logs from models and estimates cache hit rates and potential savings without exposing prompt contents. This capability allows precise tuning of cluster size and cache node capacity to avoid overprovisioning or wasted expenditure. The system manages configuration consistency across GPU servers to prevent key mismatches that could harm inference quality.

Technically, the solution requires GPU instances with Elastic Fabric Adapter (EFA) for Remote Direct Memory Access (RDMA), ElastiCache clusters on cache.i8ge.48xlarge nodes running Valkey 9.1 or later, and recent versions of LMCache and the vLLM or SGLang inference engines. This infrastructure facilitates fast KV state transfers and ensures cache coherency across distributed nodes.

Despite these advances, distributed KV caching incurs overhead for short prompts where reloading cache can be slower than local recomputation. Additionally, multi-tenant and privacy-sensitive scenarios need careful cache partitioning to avoid information leaks across users. The skill includes safeguards to alert users when caching may not be beneficial or when configuration mismatches occur.

By automating the deployment and management of distributed KV caching, AWS empowers organizations running self-hosted LLMs to reduce costly GPU cycles and latency in demanding applications. This innovation is especially relevant for AI agents that maintain very long session contexts, enabling real-time interaction efficiencies that were previously challenging at scale.

Source & further reading

Cut self-hosted LLM cost and latency with the distributed KV caching agent skill ↗

Source published 2026-11-21. Claims and limitations should be checked against the original announcement or research.