DBR 14.1 - foreachBatch in Spark Connect Shared Clusters are not supported in Unity Catalog.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-09-2023 12:10 PM
I am getting this error on DBR 14.1
AnalysisException: [UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION] The command(s): foreachBatch in Spark Connect Shared Clusters are not supported in Unity Catalog.
This is the code:
wstream = df.writeStream.foreachBatch(foreach_batch_function).trigger(processingTime='1 seconds')
query=wstream.start()
The code works for DBR 13.3 LTS
is it a bug ?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-15-2023 02:47 PM
Hi - Can you provide some more information on this?
"This limitation is specific to the version you’re using."... was this removed in 14.1 but worked in older DBR's? Is it going to be returning? It's really getting difficult to keep up with these run time changes.
