https://community.databricks.com/t5/data-engineering/how-do-i-know-if-the-number-of-files-are-causing-performance/m-p/23877#M16564 <P>I have read and heard that having too many small files can cause performance problems when reading large data sets. But how do I know if that is an issue I am facing? </P> Wed, 16 Jun 2021 16:42:52 GMT User16826992666 2021-06-16T16:42:52Z https://community.databricks.com/t5/data-engineering/how-do-i-know-if-the-number-of-files-are-causing-performance/m-p/23877#M16564 <P>I have read and heard that having too many small files can cause performance problems when reading large data sets. But how do I know if that is an issue I am facing? </P> Wed, 16 Jun 2021 16:42:52 GMT https://community.databricks.com/t5/data-engineering/how-do-i-know-if-the-number-of-files-are-causing-performance/m-p/23877#M16564 User16826992666 2021-06-16T16:42:52Z https://community.databricks.com/t5/data-engineering/how-do-i-know-if-the-number-of-files-are-causing-performance/m-p/23878#M16565 <P>Databricks SQL endpoint has a query history section which provides additional&nbsp;information to debug / tune queries. One such metric under execution details is the number of files read.<span class="lia-inline-image-display-wrapper" image-alt="query-metrics"><img src="https://community.databricks.com/t5/image/serverpage/image-id/2489i34BB779CC1E45B3F/image-size/large?v=v2&amp;px=999" role="button" title="query-metrics" alt="query-metrics" /></span>&nbsp;</P><P>For ETL/Data science workloads, you could use the Spark UI of the cluster and click on "Query Details" to get this info. </P> Fri, 18 Jun 2021 20:47:00 GMT https://community.databricks.com/t5/data-engineering/how-do-i-know-if-the-number-of-files-are-causing-performance/m-p/23878#M16565 sajith_appukutt 2021-06-18T20:47:00Z