Spark Developer Associate study plan

Five weeks at 6–8 hours a week. You need an environment where you can run PySpark against data large enough that performance is visible — architecture and tuning questions make little sense on datasets that fit in memory.

Half the exam is writing transformations, so half the plan is too.

Week 1: architecture and components (20%)

Deliberately first, because everything else depends on it.

• The execution model: driver and executors, and what each does.
• Jobs, stages and tasks — and what causes a new stage to begin.
• Partitions — the unit of parallelism, and why partition count matters.
• Narrow versus wide transformations. Narrow transformations operate within a partition; wide ones require a shuffle, moving data across the cluster. This is the single most important idea in the exam.
• Lazy evaluation — transformations build a plan, actions trigger execution.
• The Catalyst optimiser and adaptive query execution at a conceptual level.
• Caching and persistence: what they do and what they cost.

Weeks 2 and 3: the DataFrame API (30%)

The largest section, two weeks.

Week 2 — core operations:

• Creating DataFrames; reading and writing common formats.
• select, filter, withColumn, drop, alias.
• Aggregations with groupBy and agg.
• Joins — types, and which produce shuffles.
• Handling nulls, and type casting.

Week 3 — going further:

• Window functions: ranking, running totals, lag and lead.
• Complex types — arrays, structs, maps — and exploding them.
• Date and string functions.
• UDFs, and why built-in functions are almost always preferable: a Python UDF forces per-row serialisation between the JVM and Python.
• Reading a snippet and predicting its output — the core exam skill.

Week 4: Spark SQL (20%) and troubleshooting (10%)

Spark SQL:

• The same logic expressed in SQL rather than the DataFrame API.
• Temporary views and how SQL and DataFrames interoperate.
• When SQL is clearer, and that both compile to the same plan.

Troubleshooting and tuning:

• Reading the Spark UI: stages, task duration distribution, shuffle volume, spill.
• Data skew — a few tasks far slower than the rest.
• Small files and excessive partitions.
• Broadcast joins — sending a small table to every executor to avoid shuffling a large one.
• Partition count: too few underuses the cluster, too many adds overhead.

Week 5: streaming (10%), Spark Connect (5%), Pandas API (5%), then practice

• Structured Streaming: treating a stream as an unbounded table, triggers, output modes, checkpointing and why it matters for recovery.
• Spark Connect: decoupling the client from the cluster, and what that enables for deploying applications. About two questions — an hour is enough.
• Pandas API on Spark: a pandas-like interface over distributed data, for people who know pandas. Also about two questions.
• Then a full practice exam, mistakes sorted by section, weakest rebuilt, second practice exam.

Where the hours go

The habit that earns marks

For every transformation you study, ask: does this cause a shuffle?

filter, select, withColumn — no. groupBy, join, distinct, repartition — yes. That single distinction explains stage boundaries, most performance problems, and a surprising share of the architecture and tuning questions.

Run the same job on a dataset with a skewed key and watch the task duration distribution in the Spark UI. Reading about skew teaches you the word; seeing it teaches you the pattern.