DBR 14.1 - foreachBatch in Spark Connect Shared Clusters are not supported in Unity Catalog.

[samur]

I am getting this error on DBR 14.1

AnalysisException: [UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION] The command(s): foreachBatch in Spark Connect Shared Clusters are not supported in Unity Catalog.

This is the code:

wstream = df.writeStream.foreachBatch(foreach_batch_function).trigger(processingTime='1 seconds')

query=wstream.start()

The code works for DBR 13.3 LTS

is it a bug ?

[Kaniz]

Hi @samur, The error message you’re encountering in DBR 14.1 indicates that the command foreachBatch within Spark Connect Shared Clusters is not supported in the Unity Catalog. This limitation is specific to the version you’re using. If you need to work with foreachBatch in DBR 14.1, consider exploring alternative approaches or adjusting your code to avoid this specific command within the context of Spark Connect Shared Clusters in the Unity Catalog. Happy coding! 😊

[Zach-dbMotion]

Hi - Can you provide some more information on this?

"This limitation is specific to the version you’re using."... was this removed in 14.1 but worked in older DBR's? Is it going to be returning? It's really getting difficult to keep up with these run time changes.