Serving large language models (LLMs) like GLM-5.3 efficiently poses significant GPU memory and processing coordination challenges, especially as models exceed the memory capacity of single instances and handle long prompts that slow token generation. Addressing these issues, Amazon SageMaker HyperPod Inference has integrated NVIDIA Dynamo, combining Dynamo's distributed inference capabilities with HyperPod's infrastructure management.
NVIDIA Dynamo is an open-source framework designed for scalable multi-node deployments. It enables disaggregated prefill and decode phases with KV-aware routing across engines such as vLLM, SGLang, and TensorRT-LLM. Prefill workers process prompts, while decode workers generate tokens independently, connected via high-bandwidth Elastic Fabric Adapter (EFA) networking that transfers key-value caches (KV cache) efficiently.
The integration is managed through a HyperPod Inference add-on that installs custom resources including the HyperPod Inference Operator, Dynamo operator, Grove for pod coordination, and Kubernetes AI Scheduler for coordinated GPU scheduling. AWS Deep Learning Containers provide the runtime environment. Deployment configurations allow users to bring existing DynamoGraphDeployments or use a simplified InferenceEndpointConfig with dynamoSpec for streamlined endpoint management.
Dynamo's KV-aware router intelligently reuses prefix KV cache to reduce prompt recomputation and balances load among active workers. HyperPod monitors node and GPU health, manages recovery, and orchestrates scaling with auto-replacement of failed nodes, reducing operational overhead and enhancing availability. These features together enable efficient serving of models too large for single instances, including full multi-node parallelism.
Benchmarking shows substantial throughput improvements using Dynamo on HyperPod. For example, disaggregated serving of the Qwen3-235B-A22B-FP8 model with TensorRT-LLM achieved more than twice the throughput of aggregated serving under high concurrency. Multi-node aggregated serving of Qwen3.5-397B-A17B-FP8 maintained near-single-node throughput when distributed across two instances, demonstrating the platform's scalability without significant performance penalty.
Best practices for deployment emphasize selecting appropriate Dynamo recipes accounting for model size, KV cache, and runtime requirements, validating EFA communication for multi-instance setups, benchmarking with representative workloads, and iterative tuning guided by Dynamo’s optimization tools. These help optimize cost, latency, and throughput according to application needs.
While the managed integration simplifies large model inference deployment and scaling, users must still carefully size deployments and monitor workload characteristics to achieve optimal results. The approach assumes familiarity with Kubernetes, the SageMaker environment, and Dynamo configurations which could present a learning curve for new users.
Amazon SageMaker HyperPod's support for NVIDIA Dynamo represents an important advancement in managing and serving large-scale LLMs in production, combining distributed inference with automated infrastructure management to meet increasing demands for efficient, scalable AI model deployment.
Source & further reading
Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference ↗Source published 2026-11-17. Claims and limitations should be checked against the original announcement or research.