We are having some issues with merge performance, so I went and read a bit in the documentation, I found this section:
https://docs.databricks.com/delta/tune-file-size.html#autotune-file-size-based-on-workload
"Databricks recommends setting the table property delta.tuneFileSizesForRewrites to true for all tables that are targeted by many MERGE or DML operations"
My table doesn't have this attribute, I didn't define it on table creation, and neither do I see it when i use show table properties / describe table.
However, I also saw this:
"If not explicitly set, Databricks automatically detects if 9 out of last 10 previous operations on a Delta table were MERGE operations and sets this table property to true."
All of the operations on my table are either MERGE/OPTIMIZE.
1.Does these OPTIMIZE operations count as non-merge operations and therefore causes auto-detect to fail?
2.How can I know if this feature was actually used? Will I see this as a new table property on my table? or will this happen "behind the scenes" without me being able to tell?
Thank you
