cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Does Delta perform listing of data directories?

User16869510359
Esteemed Contributor

I never ran VACUUM on the Delta table. Will Delta perform direct listing on those directories, I am afraid the query performance is going to get degraded over time

How about the log directories. I have more than 100k JSON files in the log directory

1 ACCEPTED SOLUTION

Accepted Solutions

User16869510359
Esteemed Contributor

For both Data and logs, Delta does not perform listing. The transaction logs have the details of the files and the objects are directly accessed.

Similarly with in the Delta log directory listing is performed. However, it's performed using a continuation token. Hence only partial listing is done.

View solution in original post

1 REPLY 1

User16869510359
Esteemed Contributor

For both Data and logs, Delta does not perform listing. The transaction logs have the details of the files and the objects are directly accessed.

Similarly with in the Delta log directory listing is performed. However, it's performed using a continuation token. Hence only partial listing is done.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.