- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-11-2025 05:26 AM
Hello :),
I have noticed a regression in one of my job and I don't understand why.
%python
print("Hello 1")
def toto(df, _):
print("Hello 2")
spark.readStream\
.format("delta")\
.load("/databricks-datasets/nyctaxi/tables/nyctaxi_yellow")\
.writeStream\
.foreachBatch(toto)\
.trigger(availableNow=True)\
.start()\
.awaitTermination()With a Legacy 15.3 DBR cluster, both prints are displayed.
With a Unity Catalog 15.3 cluster, only the first one is displayed.
But here is what I can find in "Standard error logs" :
Streaming ForeachBatch worker Started batch 0 with DF id 61cb46fc-3c78-4647-9784-ac01...
Hello 2
Streaming ForeachBatch worker Completed batch 0 with DF id 61cb46fc-3c78-4647-9784-ac01.....
ERROR: Query termination received for [id....Same behavior for a df.show(2), the result is displayed in error logs.
Any idea why this is happening?
Thanks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-11-2025 05:56 AM - edited 08-11-2025 06:24 AM
Hi @MaximeGendre ,
Probably you hit some streaming limitations that apply to Unit Catalog standard access mode. Assuming of course you're using standard access mode 🙂
But one of the limitation they introduce at Databricks Runtime 14.0 and UC cluster is following:
Compute access mode limitations for Unity Catalog | Databricks Documentation
Which is exactly what you're experiencing. So for Unity Catalog enabled clusters and DBR >= 14.0 print within foreachbatch will write output to driver's log.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-12-2025 12:53 AM
Hi @szymon_dybczak,
thanks a lot for the quick and accurate answer 🙂
I forgot that there was this limitation.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-12-2025 01:03 AM
Hi @MaximeGendre ,
No problem, great that it worked for you 🙂