cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How can I know if databricks auto-detected to use tuneFileSizesForRewrites?

ron_lusha
New Contributor

We are having some issues with merge performance, so I went and read a bit in the documentation, I found this section:

https://docs.databricks.com/delta/tune-file-size.html#autotune-file-size-based-on-workload

"Databricks recommends setting the table property delta.tuneFileSizesForRewrites to true for all tables that are targeted by many MERGE or DML operations"

My table doesn't have this attribute, I didn't define it on table creation, and neither do I see it when i use show table properties / describe table.

However, I also saw this:

"If not explicitly set, Databricks automatically detects if 9 out of last 10 previous operations on a Delta table were MERGE operations and sets this table property to true."

All of the operations on my table are either MERGE/OPTIMIZE.

1.Does these OPTIMIZE operations count as non-merge operations and therefore causes auto-detect to fail?

2.How can I know if this feature was actually used? Will I see this as a new table property on my table? or will this happen "behind the scenes" without me being able to tell?

Thank you

1 REPLY 1

Anonymous
Not applicable

Hi @Ron Serruya​ 

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question. Thanks.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!