ExamSprintLab

Spark Developer: the DataFrame API

Updated September 20, 2026

Developing Apache Spark DataFrame/DataSet API Applications is worth 30% of the Spark Developer Associate exam — around 14 of the 45 scored questions and the largest section. All code is Python.

The exam is multiple choice, so you read PySpark rather than write it. But the fluency required is precise: you will be shown a snippet and asked what it returns, which of four versions is correct, or what is wrong with it.

Core operations

The everyday API, and the exam expects exactness:

  • select, filter / where, withColumn, withColumnRenamed, drop, alias
  • groupBy with agg, and the aggregate functions
  • orderBy / sort, limit, distinct
  • union, and why column order matters
  • Reading and writing common formats, with mode options

Precision matters. Questions distinguish between methods that look similar but behave differently — distinct() versus dropDuplicates(subset), union versus unionByName, drop on a missing column versus an existing one.

Joins

  • Inner, left, right, outer, semi and anti joins, and what each returns.
  • Join conditions, and handling duplicate column names in the result.
  • Which joins shuffle — all of them, unless one side is broadcast.
  • That a non-unique key on one side multiplies rows. This is the correctness trap that appears on every data exam.

Window functions

The most advanced API content, and reliably tested:

  • Defining a window with partitionBy and orderBy.
  • Ranking: row_number, rank, dense_rank, and how they differ on ties.
  • lag and lead for period comparison.
  • Running totals with a frame specification.

The canonical pattern to know cold: the latest row per group — rank by date descending within a partition, filter to rank 1.

Complex types

  • Arrays, structs and maps.
  • explode to turn an array into rows.
  • Accessing nested struct fields.

Null handling and types

  • isNull, isNotNull, fillna, dropna.
  • That aggregates generally ignore nulls, so an average is computed over non-null rows.
  • Casting, and what happens when a cast fails.

UDFs: usually the wrong answer

A Python UDF serialises every row between the JVM and Python. It is dramatically slower than a built-in function doing the same work.

If a question shows a Python UDF performing something a built-in function could do, replacing it is almost certainly the answer. This is the most reliably testable performance idea in the section.

Built-ins also let Catalyst optimise the plan; a UDF is opaque to it.

Sample questions

Question 1. A DataFrame of orders must be reduced to the single most recent order per customer, retaining all columns. Which approach is most appropriate?

  • A. groupBy customer and take max of the order date
  • B. A window partitioned by customer ordered by date descending, filtered to row_number = 1
  • C. dropDuplicates on the customer column after sorting by date
  • D. orderBy date descending and then limit(1)
Show answer

Answer: B

A window partitioned by customer and ordered by date descending, with row_number filtered to 1, returns exactly one complete row per customer. A groupBy with max(date) loses the other columns, distinct on customer does not select by recency, and sorting alone does not reduce to one row per customer.

Question 2. Which operation would you expect NOT to trigger a shuffle?

  • A. withColumn
  • B. groupBy
  • C. distinct
  • D. repartition
Show answer

Answer: A

withColumn computes a new column within each partition without moving data, making it a narrow transformation. groupBy, distinct and repartition all require data with matching keys to be co-located and therefore shuffle.

Question 3. A column contains arrays of tags. The requirement is one row per tag per record. Which function achieves this?

  • A. collect_list
  • B. split
  • C. flatten
  • D. explode
Show answer

Answer: D

explode turns each element of an array column into its own row, producing one row per tag per record. collect_list aggregates in the opposite direction, split converts a string to an array, and flatten reduces nested arrays by one level without creating rows.

What to practise

Write the latest-row-per-group pattern with a window function until you can do it without looking. Then take one transformation and implement it twice — once with a Python UDF, once with built-in functions — and time both on a few million rows.

The difference is large enough to make the UDF questions permanent knowledge rather than a memorised rule.