How We Cut AWS EMR Costs by 40% (~$2M/Year) by Migrating 1,100 Spark Jobs from EMR on EC2 to EMR on EKS

Before exploring the case study, consider these three questions.

1. Does Your Organization Rely Heavily on Spark ETL and Data Analytics?

If your organization processes large volumes of data using Apache Spark, infrastructure efficiency and workload optimization can have a significant impact on cloud spending.

2. Are You Still Using Spark 2.x?

Older Spark versions may limit your ability to benefit from newer performance improvements. Assessing an upgrade alongside workload optimization can help improve processing efficiency and reduce costs. Actual savings depend on your workloads and environment.

3. Are You Running Spark Jobs on EMR on EC2?

If your organization relies on Amazon EMR on EC2, migrating suitable workloads to EMR on EKS may provide opportunities for improved resource utilization, workload isolation, and infrastructure flexibility.

The potential savings depend on workload characteristics, compute pricing, architecture, and optimization efforts.

If these challenges sound familiar, the following case study explains how we modernized a large-scale data processing platform while reducing costs and improving operational efficiency.

We modernized a large-scale analytics platform processing more than 200 TB of data daily by migrating approximately 1,100 Spark workloads from Amazon EMR on EC2 to Amazon EMR on EKS.

The project combined infrastructure migration, Spark performance tuning, orchestration enhancements, and observability improvements.

Key Outcomes

• 40% reduction in processing costs, equivalent to approximately $2 million in annual savings.
• Improved Spark job performance and resource efficiency.
• Better workload isolation and scalability.
• Enhanced monitoring, logging, and debugging.
• Integration with the existing Apache Airflow orchestration ecosystem.
• Improved operational stability and maintainability.
• Better workload parallelization.

The client operated a large-scale data platform supporting analytics, reporting, and business-critical systems. Its data processing environment relied on Apache Spark for ETL workloads and Apache Airflow for orchestration.

Client Environment

The platform supported a mix of daily, hourly, three-hourly, and near-real-time workloads. As data volumes increased, the existing infrastructure became more challenging to manage efficiently.

As workload complexity and data volumes grew, the client encountered several infrastructure and operational challenges.

These included rising cloud costs, inefficient resource utilization, long-running Spark jobs, limited observability, migration constraints, and difficulties scaling during peak workloads.

1. Rising AWS EMR Infrastructure Costs

The existing architecture relied on approximately 200 EMR on EC2 clusters with autoscaling enabled.

Despite autoscaling, several factors contributed to increasing infrastructure costs:

• Long-running and continuously active clusters.
• Significant idle compute capacity during non-peak periods.
• Cluster startup delays of approximately 20 minutes.
• Overprovisioned resources to accommodate peak demand.
• Limited parallel processing during workload spikes.
• Increasing infrastructure management overhead.

These challenges highlighted the need for a more flexible execution architecture.

2. Inefficient Resource Utilization in Spark Jobs

Several Spark workloads were not optimized for efficient CPU and memory consumption.

Key issues included:

• Excessive executor allocation.
• Inefficient memory utilization.
• CPU underutilization.
• Unbalanced partitioning strategies.

These inefficiencies increased resource consumption without delivering proportional improvements in processing performance.

3. Long-Running Spark Jobs and SLA Constraints

Several ETL pipelines experienced extended execution times because of inefficient query execution and workload configuration.

The primary issues included:

• Suboptimal Spark query execution plans.
• Expensive joins and shuffle operations.
• Poor partitioning strategies.
• Redundant transformations.

Some workloads were also sensitive to cluster provisioning delays and resource contention, leading to execution delays during peak periods.

4. Operational Overhead and Limited Observability

The operations team relied primarily on CloudWatch logs, with limited centralized visibility into workload performance and failures.

The team needed better capabilities for:

• Spark job debugging and log analysis.
• Runtime observability.
• SLA monitoring and tracking.
• Cost attribution across teams.
• Failure pattern analysis.
• Resource utilization monitoring.

Limited visibility increased troubleshooting effort and made it harder to identify optimization opportunities.

5. Migration Constraints

The client wanted to modernize its data platform without disrupting existing business processes or rebuilding its Apache Airflow orchestration workflows.

The migration therefore needed to preserve existing operational patterns while introducing the new execution architecture.

6. Scalability and Parallelization Limits

The existing platform struggled to accommodate peak workloads efficiently and offered limited flexibility for running additional jobs in parallel.

A more scalable architecture was needed to improve workload isolation, resource allocation, and execution flexibility.

We implemented an ETL modernization and optimization program centered on migrating workloads from EMR on EC2 to EMR on EKS.

The engagement included:

• EMR on EC2 to EMR on EKS migration.
• Spark workload optimization.
• Infrastructure and resource tuning.
• Containerization and Docker image standardization.
• Apache Airflow integration enhancements.
• Centralized monitoring and debugging.
• Cloud cost optimization.

EMR on EKS Migration Architecture

We migrated approximately 1,100 Spark jobs to an EMR on EKS architecture designed to improve infrastructure flexibility and workload management.

The target architecture leveraged:

• Kubernetes-native scheduling and scalability.
• Dynamic resource allocation.
• Spot-based compute capacity.
• Improved workload isolation.
• More flexible infrastructure utilization.
• Containerized Spark workloads.

The migration was planned to minimize disruption to business operations while improving scalability and operational efficiency.

How We Migrated 1,100 Spark Jobs Safely

Rather than migrating every workload simultaneously, we adopted a phased migration strategy supported by pilot testing, parallel validation, and rollback procedures.

The migration began with a pilot involving approximately 50 Spark workloads.

The pilot was designed to validate:

• Functional compatibility.
• Runtime stability.
• Data consistency.
• Operational readiness.

This approach helped identify potential issues before expanding the migration to the remaining workloads.

We developed a custom validation framework that enabled Spark workloads to run on both platforms during migration.

The framework supported parallel execution, isolated production writes, and automated output verification.

Key validation measures included:

• Isolating production writes during validation.
• Comparing outputs between the existing and target environments.
• Verifying data consistency.
• Keeping existing business processes unaffected.

This enabled comprehensive validation before production cutover and supported a controlled migration process.

After the pilot was validated, the migration progressed incrementally using a team-based rollout strategy.

The approach supported:

• Controlled deployment risk.
• Faster issue identification.
• Easier stakeholder coordination.
• Reduced operational disruption.

Rollback procedures were maintained throughout the migration.

• Individual workloads could be rolled back independently.
• Entire workload groups could be reverted when necessary.
• SLA-critical processes retained fallback paths.

This helped protect business continuity during the transition.

Apache Spark Performance Tuning and Job Optimization

Although all approximately 1,100 workloads were migrated, around 250 Spark jobs were identified for targeted performance optimization.

The objective was to improve execution efficiency while reducing unnecessary compute consumption.

We analyzed long-running Spark workloads and optimized their execution plans.

Optimization areas included:

• Join optimization.
• Shuffle reduction.
• Partition tuning.
• Predicate pushdown improvements.
• Removal of redundant transformations.
• Improved caching strategies.
• Executor sizing optimization.

These changes improved execution efficiency for resource-intensive workloads.

We fine-tuned Spark resource allocation to improve CPU and memory utilization.

Key improvements included:

• Executor right-sizing.
• Dynamic allocation tuning.
• Memory optimization.
• Parallelism tuning.
• Workload-specific configuration tuning.
• Better alignment between workload demand and autoscaling.

These changes helped reduce unnecessary compute consumption while improving resource efficiency.

The following examples illustrate the reported improvements in selected workloads.

These examples demonstrate how query optimization and resource tuning improved execution times for the selected workloads.

Docker Image Standardization for Spark on Kubernetes

To support different Spark workloads and version compatibility requirements, we designed and implemented standardized Docker images.

The standardized images provided:

• Support for multiple Spark versions.
• Faster deployment cycles.
• More consistent execution environments.
• Simplified dependency management.
• Reduced operational overhead.

This helped teams onboard workloads more efficiently while maintaining compatibility across pipelines.

Airflow Integration: Custom EMR on EKS Operators Without Rewriting DAGs

The client relied heavily on Apache Airflow to orchestrate its data pipelines. Rebuilding existing workflows would have increased migration complexity and delivery risk.

To address this, we developed custom EMR on EKS job submission operators.

The operators supported:

• Native EMR on EKS Spark job submission.
• Compatibility with existing Airflow DAGs.
• Multi-environment execution.
• Parallel runs during migration.
• Retry and failure handling.
• Parameterized job execution.
• Runtime configuration management.
• Improved operational flexibility.

This reduced the need for DAG redesign and allowed the client to modernize its infrastructure while retaining its existing orchestration investments.

Spark Monitoring, Cost Attribution, and Debugging Framework

We improved operational visibility by introducing centralized monitoring, cost attribution, and debugging capabilities.

Team-level workload tagging helped improve financial visibility across the platform.

The capabilities supported:

• Cost allocation by business unit.
• Workload consumption tracking.
• Cost forecasting.
• Identification of further optimization opportunities.

This gave teams better visibility into their infrastructure consumption.

Centralized dashboards provided visibility into:

• Pipeline execution trends.
• SLA compliance.
• Failure trends.
• Resource utilization.
• Job performance and execution metrics.
• Cloud cost monitoring.

These dashboards improved operational awareness and supported faster identification of performance and reliability issues.

We developed a custom log parsing and job tracking interface to simplify troubleshooting.

The interface provided:

• Centralized log visibility.
• Failure diagnostics.
• Spark execution insights.
• Error categorization.
• Faster root cause analysis.
• An improved developer experience.

These capabilities reduced debugging effort and operational dependency on platform teams.

The modernization initiative delivered a reported 40% reduction in annual processing costs, equivalent to approximately $2 million in yearly savings.

The results came from a combination of infrastructure modernization, Spark optimization, and improved resource efficiency.

Cost Optimization Contributions

These figures represent the contribution breakdown reported for the engagement.

Performance Improvements

Spark execution plan improvements and infrastructure tuning enhanced the efficiency of critical workloads.

Key improvements included:

• Faster processing for selected long-running jobs.
• Reduced runtime for critical Spark workloads.
• Improved workload parallelism.
• Better infrastructure elasticity.
• Reduced provisioning delays and cluster contention.
• More efficient resource utilization.

Operational Stability Improvements

The modernization program also strengthened day-to-day operations.

Improvements included:

• Better monitoring and alerting.
• Enhanced SLA visibility.
• Improved failure handling.
• Faster incident response.
• Team-level cost ownership.
• Better debugging capabilities.
• More predictable workload execution.

Scalability and Future Readiness

The EMR on EKS architecture established a scalable foundation for future platform growth.

The platform was better positioned to accommodate:

• Increasing data volumes.
• More concurrent workloads.
• Onboarding across multiple teams.
• Greater platform standardization.

The project involved the following technologies and services:

• Amazon EMR on EKS.
• Amazon EMR on EC2.
• Apache Spark.
• Kubernetes and Amazon EKS.
• Apache Airflow.
• Docker.
• Cloud monitoring and logging services.

Together, these technologies supported the migration, workload optimization, orchestration, and operational monitoring initiatives.

The engagement succeeded by balancing cost efficiency with performance, reliability, and long-term maintainability.

Rather than treating cloud optimization as a simple cost-cutting exercise, the approach addressed the underlying architecture and operational challenges.

The key factors included:

• Cost efficiency: Migrating suitable workloads and improving resource utilization.
• Performance: Optimizing Spark execution plans and job configurations.
• Operational stability: Introducing stronger monitoring, validation, and rollback procedures.
• Scalability: Adopting a more flexible containerized execution architecture.
• Developer productivity: Improving debugging tools and standardizing workload deployment.
• Maintainability: Preserving existing Airflow orchestration investments and improving operational visibility.

This combination helped create a more efficient and scalable data processing environment.

By migrating Spark workloads from EMR on EC2 to EMR on EKS and optimizing the broader ETL ecosystem, we helped transform the client's existing data processing platform into a more scalable and efficient cloud-native architecture.

The initiative delivered a reported 40% reduction in annual processing costs, or approximately $2 million in annual savings, while improving selected job runtimes, resource utilization, operational visibility, and workload scalability.

The result was a stronger foundation for future data growth, more efficient Spark processing, and continued cloud cost optimization.

If your organization runs Spark on EMR on EC2, maintains legacy Spark workloads, or faces rising ETL infrastructure costs, a structured assessment can help identify opportunities for optimization.

The Inquisitive Digital team can help assess your existing architecture, identify resource inefficiencies, evaluate EMR on EKS migration opportunities, and optimize Spark workloads.

Talk to the Inquisitive Digital team to explore cloud cost optimization opportunities for your data platform.