How to get the count of files/partition for a Delta table?

[brickster_2018]

I have a delta table and I run optimize command regularly. However, I still see a large number of files in the table. I wanted to get a break up of the files in each partition and identify which partition has more files. What is the easiest way to get this information?

[brickster_2018 · Accepted solution]

The below code snippet will give details about the file count per partition

import com.databricks.sql.transaction.tahoe.DeltaLog
import org.apache.hadoop.fs.Path
 
val deltaPath = "<table_path>"
val deltaLog = DeltaLog(spark, new Path(deltaPath + "/_delta_log"))
val currentFiles = deltaLog.snapshot.allFiles
display(currentFiles.groupBy("partitionValues.col").count().orderBy($"count".desc))

[saurabh18cs]

Hi,

how to install this library 'import com.databricks.sql.transaction.tahoe.DeltaLog' in databricks cluster? as I am getting module not find error.

TQ

BR

Saurabh