topic Re: Find the size of delta table for each month before partition in Data Engineering

Find the size of delta table for each month before partition

chandraprakash — Thu, 25 Jan 2024 06:48:07 GMT

We have 38 delta tables. We decided to do partition the delta tables for each month.

But we have some small tables as well. So we need find the size of delta tables for each month. So that we can use either partition or Z-order

Is there a way to find the size of delta table for each month?

Re: Find the size of delta table for each month before partition

Lakshay — Thu, 25 Jan 2024 11:42:06 GMT

I think the only option here is to run a count by the partition column. That will give you the no. of rows for each partition.

Re: Find the size of delta table for each month before partition

dennyglee — Sat, 27 Jan 2024 07:41:27 GMT

For your tables, I’m curious if you could utilize Liquid Clustering to reduce some of the maintenance issues relating to choosing Z-Order vs. partitioning.

Saying this, one potential way is to read the Delta transaction log and read the Add Info statistics which includes the file path and size.

You can query the transaction log add info directly to extract out all the files that are associated with a particular month to calculate its size.

You can also find more info about this at A peek into the Delta Lake Transaction Log.

HTH!