NVIDIA recently announced the open source release of Run:ai, an AI workload orchestration software tailored for Kubernetes environments. A key feature, the GPU Fractioning Operator, enables multiple Kubernetes Pods to share the same NVIDIA GPU on Amazon Elastic Kubernetes Service (EKS) with enforced GPU memory contracts, overcoming limitations of previous GPU sharing approaches.
Traditional GPU resource sharing methods include Multi-Instance GPU (MIG), time-slicing, and Multi-Process Service (MPS). While MIG provides hardware-level isolation, it requires GPU support. MPS enables concurrent CUDA processes but lacks enforced memory isolation. NVIDIA Run:ai builds upon MPS, adding Kubernetes-integrated scheduling, per-Pod memory limits, and detailed metrics to ensure more efficient and controlled GPU utilization.
The validated example demonstrated on an Amazon EC2 g5.xlarge instance with an NVIDIA A10G GPU shows two Pods each allocated an 8 GiB GPU memory contract running concurrently on a single GPU with successful CUDA memory allocations and detailed per-Pod GPU memory metrics. Attempting to allocate memory exceeding the contract resulted in immediate rejection, illustrating effective enforcement.
Implementing this solution involves building a custom Amazon Linux 2023 image with the NVIDIA R615 driver branch, deploying the GPU Operator and KAI Scheduler, and configuring Pods with fractional GPU memory requests. This setup provides a refined mechanism for workload scheduling and GPU memory sharing on Amazon EKS, relying on software enforcement rather than hardware partitions.
This approach is particularly beneficial for workloads such as interactive notebooks, model evaluation tasks, and small inference services that do not require full GPU allocation and can tolerate shared compute resources. By enabling controlled GPU memory sharing, teams can improve cluster resource utilization and reduce wait times.
However, software-based GPU fractioning does not guarantee fault isolation or proportional compute allocation, and it requires careful monitoring of memory usage, driver health, and concurrent workload performance. It also demands trusted administrative control because the fractional GPU operator uses privileged containers and host integrations.
Currently, GPU fractioning requires fixed-size GPU node groups; dynamic autoscaling with fractioning-enabled GPUs is not yet supported at scale within Amazon EKS. Users must validate compatibility between their GPU hardware, drivers, and the fractioning operator before adoption and consider existing GPU sharing best practices outlined by AWS and NVIDIA.
In summary, NVIDIA Run:ai GPU Fractioning on Amazon EKS presents a flexible and enforced solution for fractional GPU sharing in Kubernetes, extending accessibility to GPUs without MIG and improving utilization efficiency. Production deployment calls for thorough testing, monitoring, and adherence to security best practices to manage limitations inherent in software-based GPU sharing.
Source & further reading
Using NVIDIA Run:ai GPU Fractioning on Amazon EKS ↗Source published 2026-11-19. Claims and limitations should be checked against the original announcement or research.