- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-09-2024 09:47 PM - edited 06-09-2024 09:49 PM
Dear @raphaelblg,
thank you for your message. From my understanding, predictive optimization requires Unity Catalog to be enabled, which is not the case for the respective workspace.
The history entry of the auto OPTIMIZE looks like this:
As you can see, the optimization has been triggered automatically, and no cluster columns are listed.
In contrast, when I manually trigger OPTIMIZE, the history entry looks like this:
As you can see, when triggered manually, OPTIMIZE clusters the table as expected, indicated by the columns listed under "clusterBy".
Regarding the spark config: "spark.databricks.delta.autoCompact.enabled" for the respective job has not been set (and hence is false). However, I set "spark.databricks.delta.optimizeWrite.enabled" to "true" to reduce the number of small files written. To my understanding, this is an on-write operation, and should not trigger auto OPTIMIZE.
Again, I can clearly correlate the onset of the auto OPTIMIZE entries in the table history with our switch from DBR 13.3 LTS to DBR 14.3 LTS.
As per your request, I verified that the table has indeed liquid clustering activated:
Best regards!