How to get the count of files/partition for a Delta table?

brickster_2018
Databricks Employee
Databricks Employee

‎06-23-2021 04:23 PM

I have a delta table and I run optimize command regularly. However, I still see a large number of files in the table. I wanted to get a break up of the files in each partition and identify which partition has more files. What is the easiest way to get this information?

brickster_2018
Databricks Employee
Databricks Employee

‎06-23-2021 04:57 PM

The below code snippet will give details about the file count per partition

import com.databricks.sql.transaction.tahoe.DeltaLog
import org.apache.hadoop.fs.Path
 
val deltaPath = "<table_path>"
val deltaLog = DeltaLog(spark, new Path(deltaPath + "/_delta_log"))
val currentFiles = deltaLog.snapshot.allFiles
display(currentFiles.groupBy("partitionValues.col").count().orderBy($"count".desc))

View solution in original post

0 Kudos

saurabh18cs
Honored Contributor III
In response to brickster_2018

‎12-07-2022 03:03 AM

Hi,

how to install this library 'import com.databricks.sql.transaction.tahoe.DeltaLog' in databricks cluster? as I am getting module not find error.

TQ

BR

Saurabh

0 Kudos