Apache Spark MasterClass Chapter 1 – Episode 3

MapReduce executes in coarse-grained phases (map -> shuffle -> reduce) and persists intermediate results to local disk between phases, which improves durability and simplifies retries but incurs I/O overhead. It excels in very large, batch-oriented workloads, stable nightly ETL, and environments where failure recovery and simplicity are prioritized. Spark represents computations as DAGs of transformations and tries to keep intermediate data in memory, writing to disk only when needed (e.g., spills, checkpoints, wide shuffles). Spark excels at iterative algorithms, interactive analytics, and pipelines with multiple dependent transformations, delivering lower latency and higher developer productivity via higher-level APIs (DataFrames, SQL, RDDs).

Shuffle redistributes data across the network so that all values for the same key end up on the same reducer (or Spark task). It’s expensive due to network I/O, sorting, serialization/deserialization, and disk spills. Mitigations: Use combiners/map-side aggregations to reduce data volume early; choose good keys to balance partitions and avoid skew; compress map outputs; increase memory to limit spills; tune the number of reducers/partitions; leverage map-side joins/broadcast joins when one side is small; use efficient encodings (e.g., Parquet/Avro) and avoid oversized objects.

Word count: Mapper emits (word, 1) per word; optional combiner sums local counts; reducer aggregates counts per word and writes (word, total). Top-K: After reducers compute (word, total), perform a second stage that groups results into a single reducer (or a small set) keeping a size-K min-heap to track the top-K counts. Alternatively, use map-side partial top-K and merge in the reducer to reduce data volume.

Skew occurs when a few keys receive disproportionately large amounts of data, creating straggler tasks and elongated job times. Handling: custom partitioners to spread hot keys, salting keys (key

Mapper outputs are written to local disk; on failure, the master reschedules the task on another worker to recompute the missing outputs from the original split. Reducers read mapper outputs from the surviving local disks; if a reducer fails, it can be restarted to re-read the intermediate data. Persisting intermediates plus deterministic tasks ensures progress despite failures.

Speculative execution launches duplicate attempts of slow (straggling) tasks on other workers; whichever finishes first commits. It’s useful when outliers are due to transient node issues. Avoid it if tasks are non-idempotent or if duplicated side effects are costly.

A combiner performs local aggregation on mapper outputs before shuffling, reducing data sent over the network. It is safe and effective only for associative and commutative operations (e.g., sum, count, min/max). It may be invoked 0, 1, or multiple times; the algorithm must not depend on invocation count.

Partitioners determine which reducer receives a given key (e.g., default hash partitioner). A good partitioner balances load across reducers and preserves correctness by ensuring all values for a key go to the same reducer. Custom partitioners can group related keys or split hot keys to mitigate skew (with post-processing to merge results if needed).

Split size affects parallelism and overhead. Small splits increase parallelism but add scheduling overhead; very large splits reduce overhead but may cause long tasks and uneven load. A common approach aligns splits with the underlying block size (e.g., HDFS block size) while tuning for workload characteristics (compression, record size).

For MapReduce, SequenceFile, Avro, or Parquet are common. Avro provides schema evolution and compact binary serialization; SequenceFile is simple and splittable; Parquet is columnar, efficient for analytical reads (more common in Spark). Choose formats that are splittable, compressible, and support your downstream engines and schema evolution needs.

When the environment lacks sufficient memory or stable Spark infrastructure; when workloads are massive batch jobs with simple transformations where disk-based robustness is preferred; when existing MapReduce code is hardened and migration risk is high; or when operational simplicity and predictable recovery outweigh latency needs.

Start by replicating outputs in a staging environment and validating with data diffs. Replace mappers/reducers with DataFrame/Spark SQL transformations; validate partitioning and ordering semantics, especially around shuffles. Introduce checkpoints and lineage. Run both pipelines in parallel (shadow mode), compare metrics and costs, then cut over gradually with clear rollback plans.

Write amplification occurs when data is written multiple times (map output, shuffle spills, reduce output). Reduce it by enabling map output compression, using combiners, tuning buffer sizes to limit spills, compacting outputs, and minimizing unnecessary intermediate stages. In Spark, caching and avoiding redundant wide shuffles further mitigate writes.

A map-side join joins a large dataset with a much smaller one by distributing (broadcasting) the small dataset to mappers so each mapper can join locally, avoiding a full shuffle. It is appropriate when the small dataset fits in memory per mapper and can be reliably distributed (e.g., via distributed cache or broadcast variables in Spark).

The user program submits a job; a master (e.g., JobTracker/YARN ApplicationMaster) schedules map tasks on workers against input splits. Mappers process records, emit (key, value) pairs, and write local intermediate files (optionally with combiners). Reducers remotely fetch and merge map outputs (shuffle and sort), aggregate by key, and write final results to the distributed filesystem. The master tracks task states and retries failures; job completion is signaled once all reduce tasks finish writing outputs.