SMART WORLDJOURNALSubmit Manuscript
← All technology news

Artificial Intelligence

Amazon SageMaker Enhances AI Inference with Adaptive Routing, Multi-Model Support, and Managed Multi-Node Serving

Amazon SageMaker’s 2026 updates introduce advanced inference recommendations, GPU partitioning via NVIDIA MIG, multi-model endpoints, and managed multi-node serving with NVIDIA Dynamo on HyperPod, aiming to optimize latency, utilization, and scalability for production AI models.

NEWS

Amazon SageMaker continues to evolve its inference capabilities in 2026, addressing real-world challenges of running large, multi-adapter AI models under varying workloads. Key launches include inference recommendations accessible across development tools, GPU partitioning to fit multiple models on single GPUs, and managed multi-node serving for models exceeding single-machine memory.

Inference recommendations, initially launched as an API, now integrate with SageMaker AI Studio, the Python SDK v3, and coding agents. These tools benchmark configurations on real NVIDIA GPU infrastructure using NVIDIA AIPerf, optimizing instance type, container, and concurrency settings for user-defined goals such as minimizing latency or cost. The system also supports tuning workloads with multiple LoRA adapters and performs SLA-based concurrency searches to find highest sustainable request loads within latency targets.

To improve GPU utilization, SageMaker introduced Inference Components with console support, allowing multiple models to run on one endpoint with individually configurable resources and autoscaling policies. This complements GPU partitioning enabled by NVIDIA Multi-Instance GPU (MIG) technology, which slices physical GPUs into isolated compute-memory partitions. Although users pay standard instance rates, this allows packing more models per GPU and better resource usage without changing client code.

For workloads requiring models too large for a single machine, Amazon SageMaker enhanced HyperPod Inference by adding managed NVIDIA Dynamo as a serving framework. Dynamo supports multi-node sharded serving, disaggregated prefill and decode stages, and KV-cache-aware routing to optimize token generation latency. Operating as a managed service, HyperPod handles GPU capacity provisioning, node recovery, autoscaling, and endpoint registration, easing operational burdens.

Routing of inference requests is further improved by KV-cache-aware routing in SageMaker AI real-time endpoints. This technique routes requests to the instance holding the most relevant cached computations, reducing redundant work for repeated or multi-turn prompts. This method requires specific verified model containers but offers better cache reuse compared to prior prefix-aware routing methods that relied solely on request history.

The Inference Gateway introduced for HyperPod clusters enables intelligent pod-level routing within a model serving cluster. It considers real-time pod states like KV cache utilization and queue depths, dynamically selecting the best pod to reduce latency and avoid idle resources. This consolidates multiple model endpoints into a single OpenAI-compatible endpoint per cluster, simplifying client integration.

Collectively, these improvements help teams customize and optimize AI inference deployments across regions, models, and workloads. They address challenges such as fitting multiple models on GPUs, serving very large models, maintaining latency targets under load, and ensuring efficient routing of requests to reduce idle resources and wait times.

Limitations include the need for AWS-verified model containers to support the most advanced routing features and the pricing structure of instances which doesn’t reduce when using MIG partitions, thus savings come from increased utilization rather than lower instance cost. The managed multi-node serving currently supports specific serving frameworks and is available only where HyperPod is offered.

Source & further reading

Amazon SageMaker Inference: 2026 launches in review, part 2 ↗

Source published 2026-11-20. Claims and limitations should be checked against the original announcement or research.