Spark Developer: architecture and components
Apache Spark Architecture and Components is worth 20% of the Spark Developer Associate exam — around 9 of the 45 scored questions. That is unusually high for an associate certification, and deliberately so: without it, the tuning and troubleshooting content is unintelligible.
Study this section first. It explains everything else.
The execution model
- Driver — runs your program, builds the execution plan, and coordinates work.
- Executors — run tasks and hold data in memory across the cluster.
- Cluster manager — allocates resources.
The practical consequence, tested repeatedly: anything that pulls data back to the driver — collect() on a large DataFrame — defeats distribution and can exhaust the driver’s memory.
Jobs, stages and tasks
The hierarchy:
- An action triggers a job.
- A job is divided into stages, and a new stage begins at a shuffle boundary.
- Each stage runs as tasks, one per partition, in parallel across executors.
So: tasks per stage = partitions. That single equation explains why partition count controls parallelism, and why too few partitions leaves your cluster idle while too many add scheduling overhead.
Narrow and wide transformations
The most important distinction in the exam.
| Narrow | Wide | |
|---|---|---|
| Data movement | Within a partition | Across the cluster |
| Shuffle | No | Yes |
| Stage boundary | No | Yes |
| Examples | filter, select, withColumn, map | groupBy, join, distinct, repartition, orderBy |
A shuffle writes data to disk and moves it across the network. It is the most expensive thing Spark does, and nearly every performance question traces back to one.
Ask of every operation: does this move data between partitions? If yes, it is wide, it causes a shuffle, and it starts a new stage.
Lazy evaluation
Transformations build a plan and do nothing. Actions — count, collect, show, write — trigger execution.
Consequences the exam tests:
- A long chain of transformations costs nothing until an action runs.
- Calling two actions on the same chain executes it twice, unless you cache.
- An error inside a transformation surfaces when the action runs, not where the code was written.
Catalyst and adaptive query execution
At the level this exam asks:
- Catalyst optimises your logical plan before execution — reordering filters, pruning columns.
- Adaptive query execution adjusts the plan at runtime using actual statistics, for example switching to a broadcast join or coalescing partitions after a shuffle.
You are not expected to tune these. You are expected to know Spark does not execute your code literally as written.
Caching and persistence
Useful when a DataFrame is reused several times and recomputation is expensive. Not useful for something read once, and over-caching consumes memory that execution needs, causing spill.
Sample questions
Question 1. A DataFrame has 8 partitions and is processed on a cluster with 32 available task slots. What limits parallelism?
- A. Executor memory
- B. The partition count — 8 partitions means 8 tasks, so 24 slots sit idle
- C. The number of cores on the driver
- D. The file format of the source data
Show answer
Answer: B
Spark runs one task per partition per stage, so 8 partitions produce 8 tasks and only 8 slots can be used regardless of the 32 available. Executor memory, driver cores and the file format do not change the task count.
Question 2. Which sequence correctly describes what happens when an action is called?
- A. A job is created, divided into stages at shuffle boundaries, and each stage runs as one task per partition
- B. A task is created, divided into jobs, each of which runs a stage
- C. A stage is created, divided into jobs, each of which runs one partition
- D. Each partition creates a job, which is divided into tasks
Show answer
Answer: A
An action triggers a job, which is split into stages at shuffle boundaries, and each stage runs as one task per partition. The other orderings invert the hierarchy or make the stage the top-level unit, which is incorrect.
Question 3. An engineer calls collect() on a DataFrame containing 800 GB of data to compute a sum. What is the problem?
- A. The source file format is inefficient
- B. The DataFrame has too many partitions
- C. collect() pulls all data to the driver; the aggregation should run distributed
- D. The DataFrame was not cached beforehand
Show answer
Answer: C
collect() returns all data to the driver, which cannot hold 800 GB and abandons distributed execution; the aggregation should be performed across executors. The issue is not the file format, not the number of partitions, and not the absence of caching.
What to practise
Run a job with a wide transformation and open the Spark UI. Count the stages and confirm the boundary falls at the shuffle. Then change the partition count and watch the task count change with it.
Fifteen minutes of that makes the architecture section concrete, and it pays back across the tuning and API sections too.