topic Re: Structure stream : difference Unity Catalog vs Legacy in Data Engineering

Structure stream : difference Unity Catalog vs Legacy

MaximeGendre — Mon, 11 Aug 2025 12:26:42 GMT

Hello :),
I have noticed a regression in one of my job and I don't understand why.

%python print("Hello 1") def toto(df, _): print("Hello 2") spark.readStream\ .format("delta")\ .load("/databricks-datasets/nyctaxi/tables/nyctaxi_yellow")\ .writeStream\ .foreachBatch(toto)\ .trigger(availableNow=True)\ .start()\ .awaitTermination()

With a Legacy 15.3 DBR cluster, both prints are displayed.
With a Unity Catalog 15.3 cluster, only the first one is displayed.

But here is what I can find in "Standard error logs" :

Streaming ForeachBatch worker Started batch 0 with DF id 61cb46fc-3c78-4647-9784-ac01... Hello 2 Streaming ForeachBatch worker Completed batch 0 with DF id 61cb46fc-3c78-4647-9784-ac01..... ERROR: Query termination received for [id....

Same behavior for a df.show(2), the result is displayed in error logs.

Any idea why this is happening?

Thanks

Re: Structure stream : difference Unity Catalog vs Legacy

szymon_dybczak — Mon, 11 Aug 2025 13:24:07 GMT

Hi @MaximeGendre ,

Probably you hit some streaming limitations that apply to Unit Catalog standard access mode. Assuming of course you're using standard access mode 🙂
But one of the limitation they introduce at Databricks Runtime 14.0 and UC cluster is following:

Compute access mode limitations for Unity Catalog | Databricks Documentation

Which is exactly what you're experiencing. So for Unity Catalog enabled clusters and DBR >= 14.0 print within foreachbatch will write output to driver's log.

Re: Structure stream : difference Unity Catalog vs Legacy

MaximeGendre — Tue, 12 Aug 2025 07:53:34 GMT

Hi @szymon_dybczak,
thanks a lot for the quick and accurate answer 🙂

I forgot that there was this limitation.

Re: Structure stream : difference Unity Catalog vs Legacy

szymon_dybczak — Tue, 12 Aug 2025 08:03:13 GMT

Hi @MaximeGendre ,

No problem, great that it worked for you 🙂