topic How to get the count of files/partition for a Delta table? in Data Engineering

How to get the count of files/partition for a Delta table?

brickster_2018 — Wed, 23 Jun 2021 23:23:30 GMT

I have a delta table and I run optimize command regularly. However, I still see a large number of files in the table. I wanted to get a break up of the files in each partition and identify which partition has more files. What is the easiest way to get this information?

Re: How to get the count of files/partition for a Delta table?

brickster_2018 — Wed, 23 Jun 2021 23:57:11 GMT

The below code snippet will give details about the file count per partition

import com.databricks.sql.transaction.tahoe.DeltaLog
import org.apache.hadoop.fs.Path
 
val deltaPath = "<table_path>"
val deltaLog = DeltaLog(spark, new Path(deltaPath + "/_delta_log"))
val currentFiles = deltaLog.snapshot.allFiles
display(currentFiles.groupBy("partitionValues.col").count().orderBy($"count".desc))

Re: How to get the count of files/partition for a Delta table?

saurabh18cs — Wed, 07 Dec 2022 11:03:35 GMT

Hi,

how to install this library 'import com.databricks.sql.transaction.tahoe.DeltaLog' in databricks cluster? as I am getting module not find error.

Saurabh