Spark Developer: SQL, tuning and streaming
Using Spark SQL (20%), Troubleshooting and Tuning (10%), Structured Streaming (10%), Spark Connect (5%) and the Pandas API (5%) are 50% of the Spark Developer Associate exam between them — around 22 of the 45 scored questions.
Using Spark SQL (20%)
The same engine, a different surface.
- Writing SQL against DataFrames registered as temporary views.
- Moving between the two:
spark.sql()returning a DataFrame, and a DataFrame registered for SQL. - Standard analytical SQL: joins, aggregations, CTEs, window functions.
The key insight the exam tests: SQL and the DataFrame API compile to the same plan. Neither is inherently faster. Choose for clarity — SQL is often clearer for complex joins and aggregations, the DataFrame API for programmatic construction.
Troubleshooting and tuning (10%)
About five questions, and they rest entirely on the architecture section.
| Problem | Symptom | Remedy |
|---|---|---|
| Data skew | A few tasks far slower than the rest | Redistribute or salt the key |
| Small files | Many tiny tasks; overhead dominates | Compact into fewer files |
| Too few partitions | Cluster underused; slots idle | Repartition upward |
| Too many partitions | Scheduling overhead exceeds work | Coalesce |
| Shuffle-heavy join | Large shuffle volume | Broadcast the small side |
| Spill | Data written to disk mid-stage | Reduce partition size; cache less |
Broadcast joins are the highest-value optimisation to know. When one side of a join is small, broadcasting it to every executor removes the shuffle of the large side entirely.
Structured Streaming (10%)
About five questions on the concepts:
- A stream is an unbounded table; queries over it produce results continuously.
- Triggers control when micro-batches run.
- Output modes — append, update, complete — and which suits which query.
- Checkpointing records progress and state so a stream can recover after failure without reprocessing or losing data. This is the most testable idea in the section.
- Watermarks for handling late data.
Spark Connect (5%)
About two questions. Spark Connect decouples the client from the cluster: your application talks to Spark over a protocol rather than running inside the driver process. That makes it easier to embed Spark in applications, use it from different environments, and upgrade client and server independently.
Know what problem it solves. Do not go deeper.
Pandas API on Spark (5%)
Also about two questions. A pandas-like interface over distributed data, letting people who know pandas work at Spark scale without rewriting in the DataFrame API. Know that it exists, who it is for, and that it is distributed underneath despite the familiar syntax.
Sample questions
Question 1. A developer wonders whether to write a complex aggregation in Spark SQL or the DataFrame API for better performance. What is the correct expectation?
- A. Spark SQL is faster because it is optimised separately
- B. The DataFrame API is faster because it avoids parsing
- C. Both compile to the same plan; choose for clarity
- D. SQL cannot express complex aggregations in Spark
Show answer
Answer: C
Both surfaces are compiled by Catalyst into the same optimised plan, so performance is equivalent and the choice should be made for readability and maintainability. Neither is inherently faster, and SQL is not limited to simple queries.
Question 2. A Structured Streaming job is restarted after a failure and reprocesses several hours of data already written downstream. What was most likely missing or misconfigured?
- A. Checkpointing
- B. The output mode
- C. The trigger interval
- D. The cluster size
Show answer
Answer: A
Checkpointing stores processing progress and state, allowing a restarted stream to resume where it stopped rather than reprocessing. Output mode controls what is emitted, trigger interval controls batch timing, and cluster size affects throughput rather than recovery position.
Question 3. In one stage, 4 of 300 tasks run for 25 minutes while the remainder finish in seconds. What is the cause?
- A. Too few partitions overall
- B. Data skew on the partitioning or join key
- C. The small file problem
- D. Insufficient driver memory
Show answer
Answer: B
A handful of very slow tasks among many fast ones indicates data skew, where certain keys hold disproportionate data. Too few partitions would slow all tasks evenly, small files produce many uniformly small tasks, and driver memory affects collection rather than task distribution.
What to practise
Register a DataFrame as a view, write the same aggregation both ways, and compare the physical plans — seeing them match settles the SQL-versus-API question permanently.
Then create a skewed key and watch the task distribution, and run a stream with and without a checkpoint location to see what recovery looks like in each case.