Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-27-2024 03:39 AM
I have logged the issue with Microsoft last week and they confirmed it is a Databricks bug. A fix is supposedly being rolled out at the moment across Databricks regions.
As anticipated, we have engaged the Databricks core team to further investigate the issue and get an update.
They have confirmed that the issue was caused by a regression in DBR14.3 and had been fixed and should be ready in all regions soon.
In meanwhile they also shared the following two configs that can be used to mitigate the issue:
spark.databricks.optimizer.estimateUnion.enabled to false
or
spark.databricks.optimizer.propagateStatsThroughWindow.enabled to false