How can I know if databricks auto-detected to use tuneFileSizesForRewrites?

ron_lusha — Sun, 04 Jun 2023 13:27:32 GMT

We are having some issues with merge performance, so I went and read a bit in the documentation, I found this section:

https://docs.databricks.com/delta/tune-file-size.html#autotune-file-size-based-on-workload

"Databricks recommends setting the table property delta.tuneFileSizesForRewrites to true for all tables that are targeted by many MERGE or DML operations"

My table doesn't have this attribute, I didn't define it on table creation, and neither do I see it when i use show table properties / describe table.

However, I also saw this:

"If not explicitly set, Databricks automatically detects if 9 out of last 10 previous operations on a Delta table were MERGE operations and sets this table property to true."

All of the operations on my table are either MERGE/OPTIMIZE.

1.Does these OPTIMIZE operations count as non-merge operations and therefore causes auto-detect to fail?

2.How can I know if this feature was actually used? Will I see this as a new table property on my table? or will this happen "behind the scenes" without me being able to tell?

Thank you

Re: How can I know if databricks auto-detected to use tuneFileSizesForRewrites?

Anonymous — Sat, 17 Jun 2023 10:02:16 GMT

Hi @Ron Serruya

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question. Thanks.

topic Re: How can I know if databricks auto-detected to use tuneFileSizesForRewrites? in Data Engineering

How can I know if databricks auto-detected to use tuneFileSizesForRewrites?

Re: How can I know if databricks auto-detected to use tuneFileSizesForRewrites?