Amazon EMR introduced a native data quality validation feature in Apache Spark jobs without additional cost, leveraging AWS Glue’s Data Quality Definition Language for defining declarative rules. It supports multiple table formats such as Apache Iceberg, Hudi, and Delta Lake, and works across EMR deployment options including EC2, EKS, and Serverless.
Users can embed validation in Spark scripts at any pipeline stage and apply over 30 rule types, including dynamic comparisons and ML-based anomaly detection. Actions like failing pipelines or quarantining records help maintain data integrity. Results detail dataset, rule, and row-level issues, enabling precise quality oversight across all AWS commercial and government regions.
Source & further reading
Amazon EMR now supports native data quality checks in Apache Spark jobs ↗Source published 2026-10-12. This report was prepared with AI assistance from the linked primary-source material. Claims and limitations should be checked against the original announcement or research.