Spark Developer Associate practice test
Twenty questions across the seven sections of the Associate Developer for Apache Spark exam, weighted roughly as the real exam is. DataFrame API work carries the largest share, so it gets the most questions here.
Developing Apache Spark DataFrame/DataSet API Applications
Question 1. Which of these operations triggers execution rather than only building the plan?
- A. select()
- B. filter()
- C. count()
- D. withColumn()
Show answer
Answer: C
count() is an action and forces execution. select, filter and withColumn are transformations that only extend the logical plan.
Question 2. A DataFrame is used in five separate actions and recomputed each time from an expensive source. What should you do?
- A. Persist or cache the DataFrame before the actions
- B. Repartition it into more partitions
- C. Convert it to an RDD
- D. Collect it to the driver
Show answer
Answer: A
Persisting the DataFrame avoids recomputing the lineage for each action. Repartitioning changes distribution but does not prevent recomputation.
Question 3. What is the risk of calling collect() on a large DataFrame?
- A. It repartitions the data unnecessarily
- B. It brings all rows to the driver and can exhaust driver memory
- C. It converts the DataFrame to a stream
- D. It disables the catalyst optimiser
Show answer
Answer: B
collect() brings all rows to the driver, which can exhaust driver memory and fail the application. It does not distribute work.
Question 4. Two DataFrames must be combined keeping only rows whose key exists in both. Which join type applies?
- A. Left outer join
- B. Full outer join
- C. Left anti join
- D. Inner join
Show answer
Answer: D
An inner join keeps only matching keys. Left and full outer joins retain non-matching rows from one or both sides.
Question 5. A column must be added whose value depends on another column, using built-in logic. Which approach is preferred?
- A. withColumn with built-in column expressions
- B. A row-at-a-time Python UDF
- C. Collecting to the driver and looping
- D. Converting to an RDD and mapping
Show answer
Answer: A
Built-in column expressions run inside the engine and are optimised. A Python UDF serialises every row and blocks many optimisations.
Question 6. You need rows from the left DataFrame that have no match on the right. Which join accomplishes this directly?
- A. Inner join
- B. Left semi join
- C. Left anti join
- D. Cross join
Show answer
Answer: C
A left anti join returns exactly the left rows without a match. A left outer join returns all left rows and requires a further null filter.
Apache Spark Architecture and Components
Question 7. What is the role of the driver in a Spark application?
- A. It stores all partitions of the data
- B. It plans the job and coordinates task execution on executors
- C. It replaces the cluster manager
- D. It performs all shuffles
Show answer
Answer: B
The driver hosts the SparkSession, plans the job and coordinates tasks across executors. It does not itself process the bulk of the data.
Question 8. Which statement about transformations and actions is correct?
- A. Transformations are lazy; an action triggers execution of the accumulated plan
- B. Transformations execute immediately and actions are lazy
- C. Both execute immediately
- D. Neither executes without caching
Show answer
Answer: A
Transformations are lazy and build a plan; an action triggers execution of that plan. Nothing is computed until an action runs.
Question 9. What is a stage boundary in a Spark job?
- A. Where a DataFrame is cached
- B. Where a UDF is defined
- C. Where a file is read
- D. Where a shuffle is required and data must be redistributed
Show answer
Answer: D
A stage boundary occurs where a shuffle is required, because data must be redistributed before the next set of tasks can run.
Question 10. Adaptive query execution changes a plan at runtime. Which problem does it most commonly address?
- A. Authentication failures
- B. Driver memory limits
- C. Poor static estimates leading to bad partition counts, join strategies and skew handling
- D. Schema evolution
Show answer
Answer: C
It uses runtime statistics to adjust partition counts, switch join strategies and handle skew, which static planning cannot do.
Using Spark SQL
Question 11. A temporary view must be visible to other sessions on the same cluster. What should you create?
- A. A standard temporary view
- B. A global temporary view
- C. A cached DataFrame
- D. A local variable
Show answer
Answer: B
A global temporary view is visible across sessions within the application’s lifetime. A standard temporary view is session-scoped.
Question 12. Which statement about the Catalyst optimiser is correct?
- A. It optimises logical and physical plans, applying rules such as predicate pushdown
- B. It only reformats SQL text
- C. It replaces the cluster manager
- D. It runs only when caching is enabled
Show answer
Answer: A
Catalyst applies rule-based and cost-based optimisation to the logical and physical plans, including predicate pushdown and projection pruning.
Question 13. A SQL query selects three columns from a Parquet table with 200 columns. What optimisation applies automatically?
- A. Broadcast join
- B. Bucketing
- C. Column pruning, reading only the selected columns
- D. Checkpointing
Show answer
Answer: C
Column pruning reads only the required columns from the columnar format, avoiding the rest entirely.
Question 14. A filter on a partition column reduces the data read dramatically. What is this called?
- A. Broadcast
- B. Caching
- C. Coalescing
- D. Partition pruning
Show answer
Answer: D
Partition pruning skips entire partitions that cannot match the filter, which is why partition columns should match common filters.
Troubleshooting and Tuning DataFrame API Applications
Question 15. In one stage, three tasks run for 40 minutes while the rest finish in seconds. What is the likely cause?
- A. Insufficient executors
- B. Data skew across partitions
- C. A missing cache
- D. Too many columns selected
Show answer
Answer: B
Data skew concentrates disproportionate data in a few partitions, so a handful of tasks dominate runtime. Adding executors does not help skewed partitions.
Question 16. A job spills heavily to disk during a shuffle. Which change is most likely to help?
- A. Tune the shuffle partition count so partitions fit in memory
- B. Cache every intermediate DataFrame
- C. Collect the data to the driver first
- D. Convert the job to use RDDs
Show answer
Answer: A
Adjusting shuffle partition count so each partition fits comfortably in memory reduces spill. Blindly caching more data worsens memory pressure.
Structured Streaming
Question 17. A streaming query must resume after a restart without reprocessing everything. What is required?
- A. A larger cluster
- B. A cached source DataFrame
- C. A checkpoint location
- D. A global temporary view
Show answer
Answer: C
A checkpoint location persists offsets and state so the query resumes where it stopped. Without it, the query restarts from the configured start position.
Question 18. Late-arriving events must still be aggregated correctly, but state cannot grow without bound. What should be configured?
- A. A larger shuffle partition count
- B. Caching the input stream
- C. A longer trigger interval
- D. A watermark on the event time column
Show answer
Answer: D
A watermark defines how late events may arrive and allows old state to be dropped, bounding memory while tolerating lateness.
Using Spark Connect to deploy applications
Question 19. What does Spark Connect change about how a client application runs?
- A. The client sends a logical plan over a connection instead of running inside the driver process
- B. It removes the need for executors
- C. It disables the Catalyst optimiser
- D. It converts all DataFrames to RDDs
Show answer
Answer: A
Spark Connect decouples the client from the cluster, so the application submits a logical plan over a connection rather than running inside the driver JVM.
Using Pandas API on Apache Spark
Question 20. Why use the pandas API on Spark rather than plain pandas for a large dataset?
- A. It produces different results that are more accurate
- B. It keeps pandas-style syntax while distributing execution across the cluster
- C. It removes the need for a cluster
- D. It converts data to RDDs for speed
Show answer
Answer: B
It keeps familiar pandas syntax while distributing execution across the cluster, so datasets larger than one machine’s memory can be processed.
How did you do?
Sixteen or more correct suggests you are close. Below fourteen, the section guides here are the fastest fix — this exam rewards knowing which operations trigger execution and which cause a shuffle. Databricks does not publish a passing score.