Amazon SageMaker Inference Enhances GPU Efficiency with Multi-Instance GPU Support

When deploying AI models to Amazon SageMaker GPU endpoints, previously each model required an entire GPU accelerator, often leading to underutilized capacity or overprovisioning. To address this, SageMaker Inference has integrated NVIDIA Multi-Instance GPU (MIG) support, which partitions a physical GPU into multiple isolated partitions, each with dedicated compute and memory resources enforced in hardware.

With MIG-enabled partitioning, a single GPU, such as the NVIDIA A100, can be divided into up to seven independent instances. Each partition guarantees predictable latency since compute and memory bandwidth are reserved, provides fault isolation to keep individual model errors contained, and offers tenant isolation for security compliance. This hardware-level separation allows multiple models or tenants to share underlying GPU resources without interference.

This upgrade enables customers to deploy more model variants than available GPUs, improving endpoint utilization. It also provides flexibility in right-sizing allocations so models needing less compute or memory can occupy a fraction of a newer GPU rather than a full older GPU, optimizing cost and performance balance. For example, small specialized models like classifiers can run on minimal partitions, while larger models receive appropriately sized slices.

Integration with SageMaker’s inference components means GPU partitioning works seamlessly with existing deployment APIs without new resource types. Customers specify partition layouts when configuring endpoints, choosing from NVIDIA’s supported MIG profiles for their instance's GPU type. However, partitions are fixed upon endpoint creation and cannot be dynamically resized without redeployment.

Performance benchmarking comparing a single H100 MIG partition to a whole A100 GPU showed the partition often outperforms the older whole GPU in throughput and latency, especially under high concurrency and long-context workloads, while also reducing cost per million output tokens by up to 34%. Nevertheless, under low concurrency or short prompts, full GPUs may still be more economical.

Limitations include a maximum of seven partitions per GPU, lack of support for serverless or heterogeneous instance pools, and that MIG partitions handle only single-GPU models. As partition types are static per model component, this feature fits stable or slowly evolving model portfolios better than dynamic workloads requiring frequent reshuffling.

Overall, SageMaker’s MIG support delivers a managed, scalable GPU partitioning solution that eliminates the complexity of self-managing Kubernetes clusters for MIG and unlocks significant efficiency gains for multi-model and multi-tenant AI inference environments.