With Spark Connect on EMR Serverless, developers can maintain a persistent Spark context across multiple notebook cells and scripts, facilitating seamless combination of local Python code execution with distributed Spark operations. Supported environments include Amazon SageMaker Unified Studio, Jupyter, and Visual Studio Code. This client-server model decouples client applications from the Spark driver, preserving developer tool preferences while offloading computation to managed EMR infrastructure.
Users benefit from real-time monitoring through the Spark UI, job history access, and centralized session management via the EMR console and APIs. The feature supports interactive data exploration, incremental debugging, and iterative Spark job development, optimizing workflows before production deployments. Availability spans all AWS Regions where EMR Serverless and SageMaker Unified Studio are offered.
Source & further reading
Run Interactive Workloads on Amazon EMR Serverless with Spark Connect ↗Source published 2026-06-09. This report was prepared with AI assistance from the linked primary-source material. Claims and limitations should be checked against the original announcement or research.