Apache Spark MasterClass Chapter 3 – Episode 5

The driver receives the action, the query plan is optimized into a DAG, and the DAGScheduler splits it into stages at shuffle boundaries. The TaskScheduler creates tasks per partition and assigns them to executors. Executors run tasks (one per core), spill/shuffle as needed, and return results. The driver aggregates stage results, marks the job complete, and the UI reflects stages/tasks timeline and metrics.

Narrow transformations (map/filter) can be pipelined within a stage because each output partition depends on one input partition. Wide transformations (groupBy/orderBy/join) require shuffles, forcing stage boundaries and expensive disk/network I/O. Minimizing wide ops, pre-partitioning data, and leveraging broadcast joins mitigate costs.

`count(‘Count’)` counts rows per (State, Color); it answers ‘how many records’. `sum(‘Count’)` sums the numeric counts per group; it answers ‘how many M&Ms’. Choose based on semantics; many public examples demonstrate both patterns.

Lazy evaluation lets Catalyst reorder/push filters and combine stages for efficiency. Use `explain(true)` to inspect logical/physical plans, check for shuffles (`Exchange`), and ensure predicate pushdown. Trigger with small actions like `show(5)` instead of `collect()` during development.

Common causes: OOM, fetch failures during shuffle, bad UDF logic, corrupted input. Spark retries failed tasks (configurable) possibly on different executors; persistent failures can fail a stage. Examine UI logs, adjust memory, fix code/data, or increase `spark.task.maxFailures` judiciously.

Client: the driver runs on the submission machine; good for interactive use where logs/UI are local. Cluster: the driver runs on a worker node; preferred for production where the submitter may disconnect. Choice affects network paths to data sources and driver memory sizing.

Aim for 2–5 cores per executor to avoid single hotspots, allocate memory to fit data/shuffle, and set partitions such that tasks take tens of seconds. Use `spark.sql.shuffle.partitions` or `repartition`/`coalesce`, and validate via the UI’s task durations and skew metrics.

Stages (to inspect shuffle boundaries and time), Storage (to verify caching), SQL tab (physical plans), and Executors (GC time, task failures). Look for skewed tasks, long shuffle reads/writes, and spilled memory.

Broadcast join sends a small table to all executors to avoid shuffling the large table. It’s effective when one side fits under `spark.sql.autoBroadcastJoinThreshold` (or is explicitly broadcast), reducing network I/O and stages.

The driver holds the SparkContext, schedules jobs, and aggregates results; executors run tasks and store data/cache. Executor failure is recoverable via task retry; driver failure aborts the application unless supervised (e.g., cluster manager restarts).

Externalize configs via args/files/env variables, avoid hard-coded paths/credentials, and use profiles per env. Keep Spark/Scala versions consistent; package dependencies with Maven Shade or sbt-assembly to avoid classpath conflicts.

Caching keeps computed partitions in memory/disk for reuse; it preserves lineage. Checkpointing truncates lineage by materializing to reliable storage, useful for very long DAGs or iterative algorithms to prevent stack/lineage blowup.

Start with a tiny deterministic CSV and hand-compute expected results; assert equality. Compare `count(‘Count’)` vs. `sum(‘Count’)` to ensure semantics. Use `explain()` to confirm only necessary shuffles, and test filters for a single state like CA.

Inference can mis-type columns or be expensive on large files; types may drift between runs. Prefer explicit schemas for stability and performance, especially for numeric aggregations.

Define dependencies in `build.sbt`, run `sbt clean package` to create a fat JAR (or use sbt-assembly). Submit with `spark-submit –class main.scala.chapter2.MnMcount`, set master/deploy mode, and tune executor resources. Verify logs/UI on the driver and configure retries/monitoring.