Auto loader

bgarcia
Databricks Partner

How to get a list of file loaded by auto loader?

handreassa
New Contributor III

If you are using the checkpointLocationoption you can read all the files that were processed by reading the rocksDB logs. Some example code to achieve that, note that you need to point to the path on the checkpoint location that you want to retrieve the loaded files list.

from glob import glob
import codecs
 
directory = "<YOUR_PATH_GOES_HERE>/sources/*/rocksdb/logs/"
for file in glob(f"{directory}/*.log"):
    with codecs.open(file, encoding='utf-8', errors='ignore') as f:
        f = f.readlines()
        print(f)

View solution in original post

Babu_spark
New Contributor II

It’s great feature and saves time