https://community.databricks.com/t5/data-engineering/does-delta-perform-listing-of-data-directories/m-p/18292#M12111 <P>For both Data and logs, Delta does not perform listing. The transaction logs have the details of the files and the objects are directly accessed. </P><P>Similarly with in the Delta log directory listing is performed. However, it's performed using a continuation token. Hence only partial listing is done. </P> Fri, 25 Jun 2021 22:39:50 GMT brickster_2018 2021-06-25T22:39:50Z https://community.databricks.com/t5/data-engineering/does-delta-perform-listing-of-data-directories/m-p/18291#M12110 <P>I never ran VACUUM on the Delta table. Will Delta perform direct listing on those directories, I am afraid the query performance is going to get degraded over time</P><P>How about the log directories. I have more than 100k JSON files in the log directory</P> Fri, 25 Jun 2021 22:38:03 GMT https://community.databricks.com/t5/data-engineering/does-delta-perform-listing-of-data-directories/m-p/18291#M12110 brickster_2018 2021-06-25T22:38:03Z https://community.databricks.com/t5/data-engineering/does-delta-perform-listing-of-data-directories/m-p/18292#M12111 <P>For both Data and logs, Delta does not perform listing. The transaction logs have the details of the files and the objects are directly accessed. </P><P>Similarly with in the Delta log directory listing is performed. However, it's performed using a continuation token. Hence only partial listing is done. </P> Fri, 25 Jun 2021 22:39:50 GMT https://community.databricks.com/t5/data-engineering/does-delta-perform-listing-of-data-directories/m-p/18292#M12111 brickster_2018 2021-06-25T22:39:50Z