Find the size of delta table for each month before partition

chandraprakash
New Contributor

‎01-24-2024 10:46 PM - edited ‎01-24-2024 10:48 PM

We have 38 delta tables. We decided to do partition the delta tables for each month.

But we have some small tables as well. So we need find the size of delta tables for each month. So that we can use either partition or Z-order

Is there a way to find the size of delta table for each month?

0 Kudos

Lakshay
Databricks Employee
Databricks Employee

‎01-25-2024 03:42 AM

I think the only option here is to run a count by the partition column. That will give you the no. of rows for each partition.

0 Kudos

dennyglee
Databricks Employee
Databricks Employee

‎01-26-2024 11:39 PM - edited ‎01-26-2024 11:41 PM

For your tables, I’m curious if you could utilize Liquid Clustering to reduce some of the maintenance issues relating to choosing Z-Order vs. partitioning.  

Saying this, one potential way is to read the Delta transaction log and read the Add Info statistics which includes the file path and size.  

You can query the transaction log add info directly to extract out all the files that are associated with a particular month to calculate its size.  

You can also find more info about this at A peek into the Delta Lake Transaction Log.  

HTH!

 

0 Kudos