Batch vs streaming: how to choose

Batch processing handles a bounded set of data in one run, then stops; streaming processes events continuously as they arrive, on data that has no end. The decision rule for exams: choose by the latency the business asks for, not by what sounds modern. If a result is needed once an hour or once a day, batch is simpler and cheaper; if the stem says “as events arrive”, “within seconds” or “near real time”, use a streaming engine; if it says “incrementally, but only run when scheduled”, the middle option (a streaming engine run in batch-like mode) is often the answer.

The choice in one table

Micro-batch is the idea that makes Spark Structured Streaming work: the engine treats the stream as an ever-growing input table and processes new rows in small batches. Apache Spark’s documentation describes it this way: at each trigger, new rows are appended to the input table and the result table is updated, and the engine “reads the latest available data from the streaming data source, processes it incrementally to update the result, and then discards the source data.” The API deliberately matches batch: you use spark.readStream and writeStream with the same DataFrame operations. That is why one codebase can serve both modes, a point that features in lakehouse and medallion designs (see medallion architecture).

What a trigger decides

In Structured Streaming the trigger controls when a micro-batch runs. With no trigger set, the engine starts the next batch as soon as the previous one finishes. A fixed interval, such as trigger(processingTime="1 minute"), runs on a schedule. trigger(availableNow=True) processes everything available at start, in as many batches as needed, and then stops. That last option is the common exam answer for “ingest new files incrementally, but run it as a nightly job to save cost”: it keeps the streaming bookkeeping (checkpoints, exactly-once) while behaving like a batch job.

Worked example: counting orders per 10 minutes

A shop sends order events with a timestamp set by the app. A dashboard needs the number of orders per 10-minute window.

counts = (events
    .withWatermark("order_time", "15 minutes")
    .groupBy(window("order_time", "10 minutes"))
    .count())

Three ideas are in those lines.

1. Event time versus processing time. order_time is when the event happened; processing time is when the engine sees it. Mobile devices and queues deliver late, so aggregating by event time gives the correct answer and aggregating by arrival time does not.
2. Windows. Spark’s window(col, "10 minutes") is a tumbling window: fixed size, no overlap, each event in one window. Adding a slide, window(col, "10 minutes", "5 minutes"), gives overlapping (hopping or sliding) windows, and session_window groups events until a gap of inactivity. Fabric eventstreams, which share their windowing functions with Azure Stream Analytics, offer tumbling, hopping, sliding, session and snapshot windows.
3. Watermark. The engine cannot keep state for every window forever. The watermark is the largest event time seen minus the delay (15 minutes here). Per the Spark documentation, data delayed by less than the threshold is never dropped, data delayed by more may or may not be processed, and a window’s state is cleaned up once the watermark passes its end. In Append output mode a window’s final count is emitted only after the watermark passes the window end, so results appear late but never change. Update mode emits revised counts as they arrive.

The trade-off to remember: a longer watermark tolerates later data and costs more state and delay; a shorter one is cheaper and drops more late events.

Checkpoints and delivery guarantees

A streaming query needs a checkpointLocation. The engine records the offset range processed in each trigger, so a restarted query resumes where it stopped instead of reprocessing hours of data. Spark states that with replayable sources and idempotent sinks, Structured Streaming can provide end-to-end exactly-once semantics; the file sink is exactly-once, while Kafka and foreachBatch sinks are at-least-once by default, so foreachBatch needs the batch id for deduplication. On Delta tables this combines with the transaction log; see Delta Lake explained.

Eventstreams in Fabric

An eventstream in Microsoft Fabric Real-Time Intelligence is a no-code way to capture events, transform them and route them. Microsoft Learn lists sources such as Azure Event Hubs, IoT Hub, Kafka, Amazon Kinesis, change-data-capture connectors and custom endpoints; no-code transformations including filter, manage fields, aggregate, group by (time windows), union, expand and join; and destinations including Eventhouse (query with KQL, see the KQL cheat sheet), Lakehouse (events converted to Delta tables), derived streams, Activator and custom endpoints. Some items are in preview, among them the SQL operator, the Spark Notebook destination and schema-aware eventstreams. The documentation lists at-least-once event delivery, a 1 MB maximum message size and up to 90 days of event retention.

Which exams test this

How exam questions are built around this choice

• Latency words decide. “Nightly”, “daily report” is batch. “Within seconds”, “as it happens” is streaming. “Alert when a threshold is crossed” suggests Eventstream with Activator.
• No-code or code? Light filtering and routing with no code is an Eventstream; complex PySpark logic with Delta output is Spark Structured Streaming; time-series querying is an Eventhouse.
• Late data. “Events arrive out of order or late” means event time plus a watermark.
• Restart without reprocessing. A checkpoint.
• Full or incremental load. Incremental loads, often with a stored high-water mark, are batch even though they are not full loads.
• Cost. Streaming keeps compute running; availableNow or a scheduled batch is cheaper when freshness is not required.

Practice questions

Go deeper

• Structured Streaming: DataFrames, windows, watermarks (Apache Spark)
• Structured Streaming getting started: processing model and fault tolerance (Apache Spark)
• Microsoft Fabric Eventstreams overview (Microsoft Learn)
• Related on this site: Spark basics, DP-700 study plan, how to study for certification exams