Logging inside foreachbatch

[AndriusVitkausk]

Im having difficulty carrying out logging inside a foreachbatch.

Seems that all logging outside the foreachbatch works as expected, but inside they are only visable from the spark UI in driver logs. Is there a way to get this to work (inc serverless)?

[Ashwin_DSA]

Hi @AndriusVitkausk,

This is expected behaviour on Databricks. Code inside foreachBatch runs in the background as part of the long‑lived streaming query, so print() and standard logging go to the driver logs / Spark UI, not back to the notebook output. In Databricks Runtime 14.0+, this is explicitly documented: “print() commands write output to the driver logs” for foreachBatch on compute with standard access mode.

There isn’t a switch to redirect those logs into the notebook, including on serverless. If you need visibility outside the driver logs, Databricks recommends using structured approaches such as writing metrics or diagnostics to a Delta/UC table from inside foreachBatch, or using StreamingQueryListener + observable metrics to push per‑microbatch metrics to external monitoring systems. You may find this useful.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.