I never ran VACUUM on the Delta table. Will Delta perform direct listing on those directories, I am afraid the query performance is going to get degraded over time
How about the log directories. I have more than 100k JSON files in the log directory
For both Data and logs, Delta does not perform listing. The transaction logs have the details of the files and the objects are directly accessed.
Similarly with in the Delta log directory listing is performed. However, it's performed using a continuation token. Hence only partial listing is done.
