Spark Connect decouples the application client from the Spark driver, allowing users to maintain development environments such as Jupyter or Visual Studio Code while executing Spark workloads on Amazon EKS-managed virtual clusters. Persistent Spark contexts span notebook cells and scripts, facilitating incremental development and ad hoc data exploration workflows.
Each interactive session runs as pods with AWS Identity and Access Management (IAM) security roles, supporting project and user tagging. This feature is available with EMR releases 7.14 and emr-spark-8.1.0 across all AWS Commercial regions, integrating with Amazon SageMaker Unified Studio for managed notebook experiences.
Source & further reading
Run interactive workloads on Amazon EMR on EKS with Spark Connect ↗Source published 2026-09-24. This report was prepared with AI assistance from the linked primary-source material. Claims and limitations should be checked against the original announcement or research.