How do I know if the number of files are causing performance issues?

User16826992666
Databricks Employee
Databricks Employee

‎06-16-2021 09:42 AM

I have read and heard that having too many small files can cause performance problems when reading large data sets. But how do I know if that is an issue I am facing?

0 Kudos

sajith_appukutt
Databricks Employee
Databricks Employee

‎06-18-2021 01:47 PM

Databricks SQL endpoint has a query history section which provides additional information to debug / tune queries. One such metric under execution details is the number of files read.query-metrics 

For ETL/Data science workloads, you could use the Spark UI of the cluster and click on "Query Details" to get this info.

Preview file
44 KB
Preview file
24 KB
0 Kudos