topic Re: foreachBatch doesn't work in structured streaming in Data Engineering

foreachBatch doesn't work in structured streaming

drag7ter — Tue, 27 Aug 2024 14:05:24 GMT

I' m trying to print out number of rows in the batch, but seems it doesn't work properly. I have 1 node compute optimized cluster and run in notebook this code:

# Logging the row count using a streaming-friendly approach def log_row_count(batch_df, batch_id): display(batch_df) row_count = 0 if not batch_df.isEmpty(): row_count = batch_df.count() print(f"{row_count} rows have been appended to {FULL_TABLE_NAME}") LOGGER.info(f"{row_count} rows have been appended to {FULL_TABLE_NAME}") # Configure Auto Loader to ingest JSON data to a Delta table ptv = spark.readStream \ .format("cloudFiles") \ .option("cloudFiles.format", "parquet") \ .option("cloudFiles.schemaLocation", checkpoint_path) \ .option("cloudFiles.inferColumnTypes", "true") \ .option("cloudFiles.schemaEvolutionMode", "addNewColumns") \ .load(file_path) ptv \ .writeStream \ .option("checkpointLocation", checkpoint_path) \ .outputMode("append") \ .trigger(availableNow=True) \ .foreachBatch(log_row_count) \ .toTable(FULL_TABLE_NAME) \ .awaitTermination()

The only output I get is:

What should I do to print count in foreachBatch() ?

Re: foreachBatch doesn't work in structured streaming

Sidhant07 — Mon, 09 Dec 2024 08:56:53 GMT

Hi,

Can you try this:

def log_row_count(batch_df, batch_id): row_count = batch_df.count() print(f"Batch ID {batch_id}: {row_count} rows have been processed") LOGGER.info(f"Batch ID {batch_id}: {row_count} rows have been processed") ptv.writeStream \ .option("checkpointLocation", checkpoint_path) \ .outputMode("append") \ .trigger(availableNow=True) \ .foreachBatch(log_row_count) \ .start() \ .awaitTermination() # Write to table separately if needed ptv.writeStream \ .option("checkpointLocation", f"{checkpoint_path}_table") \ .outputMode("append") \ .trigger(availableNow=True) \ .toTable(FULL_TABLE_NAME) \ .awaitTermination()

Re: foreachBatch doesn't work in structured streaming

saffovski — Mon, 18 Aug 2025 15:22:14 GMT

Hi, I am facing the exact same error. The method that I'm calling in the foreachBatch is just a very simple print statement that test whether the method is called or no, and the print is not printed out. Here's a code snippet:

def debug_batch(batch_df, batch_id):

print(f"Batch {batch_id} started, row count = {batch_df.count()}")

query = (

df.writeStream

.option("checkpointLocation", checkpoint_path)

.trigger(availableNow=True)

.foreachBatch(debug_batch)

.start()

.awaitTermination()

)

Re: foreachBatch doesn't work in structured streaming

szymon_dybczak — Mon, 18 Aug 2025 15:48:01 GMT

Hi @saffovski ,

This is expected behavior. By default if you use print in foreachbatch it will output to driver log. So, check your driver logs 🙂

Re: foreachBatch doesn't work in structured streaming

szymon_dybczak — Tue, 19 Aug 2025 09:29:51 GMT

Hi @Advika ,

Could you mark above reply as an answer to the thread? This question keeps popping up and it would be good to have a solution for it. The behaviuor that I mentioned above is described here:

Use foreachBatch to write to arbitrary data sinks | Databricks Documentation

Re: foreachBatch doesn't work in structured streaming

Advika — Tue, 19 Aug 2025 09:57:17 GMT

Thanks for clarifying and sharing the doc @szymon_dybczak!
I’ve marked your reply as a solution.

Re: foreachBatch doesn't work in structured streaming

szymon_dybczak — Tue, 19 Aug 2025 09:59:25 GMT

Thanks @Advika 👍

Re: foreachBatch doesn't work in structured streaming

saffovski — Mon, 25 Aug 2025 14:30:54 GMT

Hi Szymon, thank you for your answer. However, if I use the froeachBatch for my actual work (reading each microbatch while reading a stream of files), it clearly shows that the foreachBatch part is not executed. I was reading something that it might be related to a serverless compute, but wasn't able to confirm this in the official Dabricks documentation.
Is this the reason?

Re: foreachBatch doesn't work in structured streaming

Malthe — Wed, 11 Mar 2026 07:47:13 GMT

@szymon_dybczak in my testing, the print output does not appear anywhere. There is no trace of them anywhere, neither in the notebook or in driver logs.