Apache Spark 4.0.2 on EMR introduces ANSI SQL support and VARIANT data types, facilitating easier development and handling of semi-structured data such as JSON without requiring specialized Spark syntax. This broadens accessibility for data engineers using Spark for large-scale data pipelines. Strengthened security features now allow fine-grained access control at row and column levels, tied to AWS Lake Formation registries, aiding in compliance and governance.
Additionally, EMR integrates Apache Iceberg version 3 table format, enhancing transactional guarantees and lineage tracking critical for audit trails. Improved streaming controls simplify stateful operation management, enabling faster deployment of latency-sensitive applications like fraud detection and personalization. This Spark version rollout is available across all Amazon EMR-supported Regions.
Source & further reading
Amazon EMR now supports Apache Spark 4.0.2 in general availability ↗Source published 2026-06-09. This report was prepared with AI assistance from the linked primary-source material. Claims and limitations should be checked against the original announcement or research.