- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-07-2024 07:27 AM - edited 06-07-2024 07:28 AM
Hello @felix_counter ,
It seems you're referring to Predictive optimization for Delta Lake, a relatively new feature.
In contrast to Optimized writes for Delta Lake on Databricks (basically `spark.databricks.delta.autoCompact.enabled ` and `spark.databricks.delta.optimizeWrite.enabled`), which apply optimizations during task write time and are not recorded in the Delta Logs, predictive optimization initiates individual OPTIMIZE operations that are indeed logged. However, it's worth noting that predictive optimization does not run OPTIMIZE operations on tables using liquid clustering or Z-order.
Are you sure that your table uses liquid clustering? If so can you please provide the results of the following statements?
DESCRIBE FORMATTED your_table_name
SHOW CREATE TABLE your_table_name
Raphael Balogo
Sr. Technical Solutions Engineer
Databricks