Hello,

I have a structured stream job writing every 5 mins into a table with liquid clustering enabled. After migrating from DBR 13.3 LTS to DBR 14.3 LTS I observe that the table is newly regularly optimized despite I have not set the "spark.databricks.delta.autoCompact.enabled" option. Furthermore, the "operationParameters" column of the table history exhibits for the auto optimize step the configuration "clusterBy":"[]", although I have specified cluster columns on this table. Does that mean the table has been (auto) optimized without triggering clustering?

Is there an option to prevent DBR 14.3 LTS from auto optimizing (liquid clustered) tables? I would prefer running OPTIMIZE (and thus triggering clustering) on a regular basis manually.

I tried setting "spark.databricks.delta.autoCompact.enabled" to "false", but that did not prevent regular auto-optimization using DBR 14.3 LTS.

thanks a lot for your help!