Databricks compute types: all-purpose, job, SQL
Databricks offers three families of compute: serverless compute, classic compute (all-purpose and job clusters that you configure), and SQL warehouses. The decision rule for exams: SQL queries and BI go to a SQL warehouse, unattended scheduled work goes to serverless or job compute, and all-purpose compute is for interactive development. If a question puts an all-purpose cluster behind a production job, it is usually the wrong answer.
Cluster or compute?
Older material says “cluster”. Current documentation says “compute” in the UI, while the APIs, the Clusters API and settings such as cluster_id keep the old word. Azure Databricks documentation lists the families as serverless compute, classic compute and SQL warehouses. Within classic compute, the configuration reference still talks about all-purpose and job compute, and job compute can be configured as classic jobs compute. The access modes were renamed too: shared became standard and single user became dedicated.
The choice, workload by workload
| Workload | Best fit | Why |
|---|---|---|
| Analyst notebook, interactive exploration | All-purpose compute, or serverless notebook compute | Stays up between commands; serverless needs no cluster management |
| Scheduled notebook, Python script or wheel | Serverless jobs (recommended); classic jobs compute also supported | Created for the run, released afterwards |
| Scheduled SQL task or dbt project | Serverless SQL warehouse (recommended); pro also supported | SQL tasks run against a warehouse |
| Dashboards and ad hoc SQL for many users | SQL warehouse, ideally serverless | Built for concurrent SQL and BI, scales with demand |
| Declarative pipeline | Serverless pipeline (recommended); classic pipeline compute also supported | Databricks recommends serverless for new pipelines |
| JAR task or Spark Submit | Classic jobs compute | Serverless is not the recommended option for these tasks |
This mapping follows the Databricks table of recommended compute per task type. Running production jobs on all-purpose compute is possible, but Databricks does not recommend it.
How the families differ
Serverless compute is on-demand, automatically managed compute for notebooks, jobs and pipelines. It has limitations: for jobs, continuous scheduling works only with bounded Structured Streaming triggers such as Trigger.AvailableNow, and default or time-based interval triggers are not supported. A question about an always-on streaming query with a processing-time trigger therefore points away from serverless jobs.
Classic compute is provisioned in your cloud subscription and you configure it: Databricks Runtime version, worker node type, autoscaling between a minimum and maximum number of workers, automatic termination after an inactivity period, spot instances, tags and an access mode. Photon is enabled by default on runtimes from 9.1 LTS upward. A cluster has one driver and zero or more workers; Spark commands need at least one worker, and a single-node cluster runs Spark locally on the driver. Instance pools keep pre-configured instances ready to cut startup time.
Job compute is classic compute defined in a job. Tasks in one job can share it; it starts with the first task that uses it and terminates after the last. A shared job cluster is scoped to one job run, so it cannot be used by other jobs or by other runs of the same job.
SQL warehouses come in three types:
| Type | Photon | Predictive IO | Intelligent Workload Management | Typical start |
|---|---|---|---|---|
| Serverless | Yes | Yes | Yes | Seconds (typically 2 to 6) |
| Pro | Yes | Yes | No | Minutes (typically about 4) |
| Classic | Yes | No | No | Minutes (typically about 4) |
Pro and classic warehouses run in your own subscription. Use pro when serverless is not available in the region, or when you need custom networking to reach databases in your own network. In the UI the default for a new warehouse is serverless where it is supported, but the SQL warehouses API defaults to classic unless you set the parameters, so exam answers that mention API creation need care.
Worked example: four tickets, four choices
A team has four requests. A nightly ETL notebook that runs for 40 minutes: serverless jobs, or classic job compute if serverless limitations rule it out, because nobody is attached to it and the compute can disappear when the run ends. Forty analysts who refresh dashboards at 9:00: a serverless SQL warehouse, which absorbs the concurrency spike without anyone sizing a cluster. A data scientist prototyping a model with a GPU: all-purpose compute with an ML runtime, which stays interactive. A Spark Submit job packaged by another team: classic job compute, since that task type is not on the serverless list. The access mode is set separately: standard lets several users share isolated compute, dedicated assigns it to one user or group, and the automatic setting picks standard unless an ML runtime, a GPU instance or a runtime below 14.3 forces dedicated.
Which exams test this
| Exam | How it appears | Where on this site |
|---|---|---|
| DP-750 | Set up and configure the environment (15-20%): choose a compute type (job, serverless, warehouse, classic, shared); configure size, autoscaling, termination, node type and pooling; Photon and runtime; libraries; permissions | Environment and compute |
| Databricks Data Engineer Professional | Developing Code (23%): choose compute and configuration, including serverless environments, dependency management, performance mode and high-memory notebook tasks | Developing code |
| Databricks Data Engineer Associate | Platform (6%): choose the most suitable compute service by characteristics, limits and cost model; Troubleshooting (10%): cluster startup failures | Platform and ingestion touches compute only briefly |
| Databricks Data Analyst Associate | Executing queries (20%): the role a SQL warehouse plays; Genie spaces choose a SQL warehouse | Executing queries |
| Databricks Spark Developer Associate | Minor: cluster, driver and worker architecture and deployment modes, which is Spark architecture rather than compute choice | Architecture explained |
How the questions are built
The scenario states who uses the compute and when. “Unattended”, “scheduled” and “nightly” point to job or serverless compute; “many concurrent analysts” or “dashboards” point to a SQL warehouse; “interactive” and “notebook development” point to all-purpose. Cost words matter: leaving an all-purpose cluster running between runs is the expensive wrong answer, and automatic termination is the usual fix. “No infrastructure management” and “fast start” point to serverless. A trap is a larger cluster offered for a problem that is really the wrong compute type, or an all-purpose cluster for a production job. For work that runs on this compute see Lakeflow Declarative Pipelines, Spark basics and batch vs streaming; access control sits in Unity Catalog.
Question 1. Forty analysts open the same dashboards at the start of the working day and run ad hoc SQL during the day. The platform team wants minimal tuning and compute that scales with concurrent queries. What should they use?
- A. A dedicated all-purpose cluster with a fixed number of workers
- B. Job compute shared across tasks
- C. A serverless SQL warehouse
- D. A single-node cluster with a large driver
Show answer
Answer: C
A serverless SQL warehouse is built for concurrent SQL and BI workloads, starts in seconds and scales with demand without sizing a cluster. An all-purpose cluster assigned to one user does not serve forty people well and sits idle. Job compute exists for the duration of a job run and cannot back interactive dashboards. A single-node cluster cannot scale with concurrency.
Question 2. A team runs a production ETL notebook every night at 02:00 and currently triggers it on a large all-purpose cluster that stays up all day. They want to cut cost without changing the code. What is the best change?
- A. Increase the all-purpose cluster size so the job finishes sooner
- B. Run the notebook as a job on serverless compute or classic job compute
- C. Move the notebook to a pro SQL warehouse
- D. Change the cluster access mode from standard to dedicated
Show answer
Answer: B
Scheduled, unattended work belongs on serverless or job compute that is created for the run and released afterwards, and Databricks does not recommend all-purpose compute for production jobs. Raising the cluster size makes the idle time more expensive. A SQL warehouse is for SQL, not for a notebook with Python logic. Switching the access mode to dedicated changes who can use the cluster, not what it costs.
Go deeper
- Compute overview (Microsoft Learn)
- Configure compute for jobs, including the recommended compute per task type
- SQL warehouse types
- Compute configuration reference
- Related on this site: the DP-750 exam tips and how to study for certification exams