Find the size of delta table for each month before partition

[chandraprakash]

We have 38 delta tables. We decided to do partition the delta tables for each month.

But we have some small tables as well. So we need find the size of delta tables for each month. So that we can use either partition or Z-order

Is there a way to find the size of delta table for each month?

[Lakshay]

I think the only option here is to run a count by the partition column. That will give you the no. of rows for each partition.

[Kaniz_Fatma]

Welcome, and thank you for being a part of our community! 

We highly value your inquiries, and we applaud your active participation. We kindly invite you to take some time to review the responses provided and select the solution that best suits your needs. Not only will you gain access to the most pertinent information, but you will also be contributing to the wealth of knowledge within the community. 

To show our gratitude, we will be awarding a special badge to users who consistently engage in selecting solutions. 

Thank you for being an integral part of our community, and please do not hesitate to reach out if you require any further assistance!

[dennyglee]

For your tables, I’m curious if you could utilize Liquid Clustering to reduce some of the maintenance issues relating to choosing Z-Order vs. partitioning.  

Saying this, one potential way is to read the Delta transaction log and read the Add Info statistics which includes the file path and size.  

You can query the transaction log add info directly to extract out all the files that are associated with a particular month to calculate its size.  

You can also find more info about this at A peek into the Delta Lake Transaction Log.  

HTH!

[Kaniz_Fatma]

Hey there! Thanks a bunch for being part of our awesome community! 🎉 

We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution for you. And remember, if you ever need more help , we're here for you! 

Keep being awesome! 😊🚀