raphaelblg
Databricks Employee
Databricks Employee

Hello @felix_counter ,

It seems you're referring to Predictive optimization for Delta Lake, a relatively new feature.

In contrast to Optimized writes for Delta Lake on Databricks (basically `spark.databricks.delta.autoCompact.enabled ` and `spark.databricks.delta.optimizeWrite.enabled`), which apply optimizations during task write time and are not recorded in the Delta Logs, predictive optimization initiates individual OPTIMIZE operations that are indeed logged. However, it's worth noting that predictive optimization does not run OPTIMIZE operations on tables using liquid clustering or Z-order.

Are you sure that your table uses liquid clustering? If so can you please provide the results of the following statements?

DESCRIBE FORMATTED your_table_name

SHOW CREATE TABLE your_table_name

 

 

 

 

Best regards,

Raphael Balogo
Sr. Technical Solutions Engineer
Databricks