topic DBR 14.1 - foreachBatch in Spark Connect Shared Clusters are not supported in Unity Catalog. in Data Engineering

DBR 14.1 - foreachBatch in Spark Connect Shared Clusters are not supported in Unity Catalog.

samur — Thu, 09 Nov 2023 20:10:33 GMT

I am getting this error on DBR 14.1

AnalysisException: [UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION] The command(s): foreachBatch in Spark Connect Shared Clusters are not supported in Unity Catalog.

This is the code:

wstream = df.writeStream.foreachBatch(foreach_batch_function).trigger(processingTime='1 seconds')

query=wstream.start()

The code works for DBR 13.3 LTS

is it a bug ?

Re: DBR 14.1 - foreachBatch in Spark Connect Shared Clusters are not supported in Unity Catalog.

Anonymous — Fri, 15 Dec 2023 22:47:27 GMT

Hi - Can you provide some more information on this?

"This limitation is specific to the version you’re using."... was this removed in 14.1 but worked in older DBR's? Is it going to be returning? It's really getting difficult to keep up with these run time changes.