Spark basics: transformations, actions, shuffles

Apache Spark is an engine that splits a dataset into partitions and processes them in parallel across a cluster. The decision rule for exams: transformations are lazy and only describe the work; an action triggers execution; and any operation that must regroup rows across partitions (a join, a group-by, a repartition) causes a shuffle, which is the expensive part. Nearly every Spark question, from the Databricks Spark Developer exam to Fabric and Databricks data engineering, comes down to spotting which of those three things is happening.

Trace one query from code to tasks

The clearest way to learn this is to follow a single piece of PySpark through the engine.

orders = spark.read.parquet("/data/orders")          # step 1
big    = orders.filter(orders.amount > 100)          # step 2
by_cust = big.groupBy("customer_id").count()         # step 3
by_cust.write.mode("overwrite").parquet("/out/cust") # step 4

Steps 1 to 3 run nothing on the data. Reading, filter and groupBy().count() are transformations. Spark’s programming guide states that transformations are lazy: they “just remember the transformations applied to some base dataset” and are only computed when an action requires a result. Variables like big are plans, not data. (Strictly, a read may touch file metadata to find the schema, which is why a bad path can fail early, but no rows are processed.)

Step 4 is the action. Writing is an action, like count(), collect(), show() and take(). Now Spark builds one job. According to the Spark cluster overview, a job is the parallel computation spawned in response to an action, made of stages, which are made of tasks, and a task is one unit of work sent to one executor.

The job splits into two stages. filter works on each partition independently, so it needs no data from other partitions. groupBy does: all rows for one customer_id must end up together. Moving them is the shuffle, and the shuffle is the boundary between stage 1 and stage 2.

Tasks follow partitions. If the Parquet read produces 200 partitions, stage 1 runs 200 tasks. The driver hands tasks to executors, which run them in parallel up to the number of available task slots. So 8 partitions on a cluster with 32 slots still means 8 tasks and 24 idle slots; parallelism is capped by partitions, not by cluster size.

Narrow and wide transformations

Spark’s guide describes the shuffle as the mechanism for re-distributing data so that it is grouped differently across partitions, typically copying data across executors and machines, which makes it “a complex and costly operation”. It involves disk I/O, serialization and network I/O. That is why tuning advice repeats the same points: filter early so fewer rows reach the shuffle, broadcast a small table instead of shuffling a large one into a join, and avoid unneeded repartition calls. Note that coalesce can reduce partitions without a full shuffle in its common use, while repartition always redistributes. A shuffle also leaves the order of rows undefined, so sort explicitly when order matters.

Why laziness matters in practice

Lazy evaluation has consequences that exam stems love:

• A long chain of transformations costs nothing until an action runs.
• Each action recomputes the whole chain unless the data is cached. The Spark guide says that by default each transformed dataset may be recomputed every time you run an action on it. Calling cache() or persist() before several actions avoids that. Caching helps when the same intermediate result is reused; it hurts when it is used once.
• A bug in a transformation surfaces when the action runs, not on the line you wrote.
• collect() pulls everything to the driver. On a very large result it is the wrong action; use write, take or an aggregation instead.

The DataFrame API and Spark SQL run on the same engine. Spark’s SQL guide says the same execution engine is used regardless of which API or language expresses the computation, so a DataFrame query and the equivalent SQL statement get the same optimisations. Spark also adapts at run time: adaptive query execution can change the plan using real statistics, for example switching to a broadcast join or merging small partitions after a shuffle. If a job is slow because a few tasks run far longer than the rest, the cause is usually data skew on the key, not the cluster size.

Which exams test this

How exam questions are built around Spark

The usual setup is a short code snippet and a question about what runs, when, or why it is slow.

• “Which line triggers execution?” Only actions do. Pick count, show, collect, write, not filter or join.
• “How many stages?” Count the wide operations; each shuffle starts a new stage. Tasks per stage equal partitions.
• “The job is slow, a few tasks take much longer.” Skew. “All tasks are slow and there are thousands of tiny files” points to small files instead.
• “The same DataFrame is used by three actions.” Cache it.
• “Reduce data movement.” Filter early, broadcast the small side of a join.

Constraint words: triggers, lazily, shuffle, stage boundary, without moving data across the network, driver memory. Streaming adds another layer on the same engine; see batch versus streaming. Table-format questions that sit under Spark are in Delta Lake explained, and pipeline layering is in medallion architecture.

Practice questions

Go deeper

• RDD programming guide: transformations, actions, shuffle, persistence (Apache Spark)
• Cluster mode overview: driver, executors, jobs, stages, tasks (Apache Spark)
• Spark SQL, DataFrames and Datasets guide (Apache Spark)
• Related on this site: Spark Developer Associate study plan, Unity Catalog explained, how to study for certification exams