Apache Spark MasterClass Chapter 3 – Episode 4

Read lines with `spark.read.text` (one row per line). Tokenize via `split` into arrays. Clean with `lower`, `regexp_extract/replace`, and `trim`. Explode arrays so each token becomes its own row, then `groupBy(‘word’).count()` and `orderBy(desc(‘count’))`. Bugs typically come from tokenization (e.g., hyphenation, punctuation, Unicode), cleaning that removes meaningful tokens, and forgetting to filter blanks/nulls.

Lazy evaluation defers execution, allowing Catalyst to optimize and minimize I/O/shuffles—great for large, chained transformations. Eager eval (REPL display) is convenient for learning/exploration but risks triggering heavy jobs inadvertently. Use eager only in small demos; prefer lazy in production.

Use `spark.read.text()` to parallelize across files, avoid UDFs, and rely on built-ins (`split`, `lower`, `regexp_replace`). Repartition to match cluster parallelism, consider stopword removal, and use `groupBy(‘word’).count()` followed by `orderBy(desc(‘count’))`. Cache intermediate cleaned tokens only if reused (e.g., multiple top-N variants).

Adjust the regex to include characters you want to keep, e.g., `regexp_extract(col(‘w’), “[a-z’]+”, 0)` to keep apostrophes or use Unicode classes like `\p{L}`. Alternatively, use `regexp_replace` to strip disallowed characters while preserving desired ones.

Built-in functions are compiled into Spark’s optimized plans and avoid Python ↔ JVM serialization overhead. UDFs disable many optimizations and are slower; use them only if no native expression covers the need (or consider pandas UDFs with care).

Massive row explosion can create data skew and memory pressure. Mitigations: filter early, tokenize carefully (e.g., ignore blank tokens), limit to necessary columns, and monitor partition sizes; use `explode_outer` for null-safe behavior.

Create small fixtures and assert: (1) no blank tokens remain after cleaning, (2) token counts for known inputs (e.g., ‘a a b’ → a:2, b:1), (3) schema expectations (array → explode → string), and (4) top-N results are stable for deterministic inputs.

`alias` within `select` is concise when you create/transform a column. `withColumnRenamed` is handy to rename an existing column after the fact or in multi-step pipelines. Both avoid fragile string parsing of generated column names.

Use `isin` with a broadcasted list or join against a small stopword DataFrame and filter. Normalize case first (`lower`), and ensure trimming; optionally keep domain-specific terms by maintaining a curated allowlist.

Use `show()` with limits, `limit()` before actions, sample with `sample()` or `take()` small counts, and inspect via the Spark UI. Avoid `collect()` on large frames; prefer `write` to storage if you need full outputs.

Compute counts once, store as a DataFrame, then call `.orderBy(desc(‘count’)).limit(n)` where `n` is a function parameter or widget input. This avoids recomputation and keeps the logic DRY.

`regexp_extract(col, pattern, group)` returns a captured substring; e.g., `[a-z]+` to keep letters. `regexp_replace(col, pattern, replacement)` substitutes matches; e.g., replace `[^a-z]` with ” to strip non-letters while preserving the rest of the string.

Read multiple files and add `input_file_name()` to keep source context, add `monotonically_increasing_id()` for line IDs, or maintain paragraph indices. Aggregate per file/section, then union into global counts for comparative analytics.

`spark.sql.shuffle.partitions` for downstream aggregations, `repartition/coalesce` for balancing, broadcast thresholds for joins (if stopword table), and caching policy. Monitor shuffle spill, skew, and executor memory in the Spark UI.

Use Unicode-aware patterns like `\p{L}+` instead of `[a-z]+`, or normalize with `unaccent` libraries before cleaning. Validate with multilingual fixtures to ensure tokens aren’t lost.