Auto loader

bgarcia — Wed, 29 Jun 2022 20:08:58 GMT

How to get a list of file loaded by auto loader?

Re: Auto loader

handreassa — Wed, 29 Jun 2022 20:20:07 GMT

If you are using the checkpointLocationoption you can read all the files that were processed by reading the rocksDB logs. Some example code to achieve that, note that you need to point to the path on the checkpoint location that you want to retrieve the loaded files list.

from glob import glob
import codecs
 
directory = "<YOUR_PATH_GOES_HERE>/sources/*/rocksdb/logs/"
for file in glob(f"{directory}/*.log"):
    with codecs.open(file, encoding='utf-8', errors='ignore') as f:
        f = f.readlines()
        print(f)

Re: Auto loader

Babu_spark — Thu, 29 Jun 2023 19:21:11 GMT

It’s great feature and saves time

topic Auto loader in Data Engineering

Auto loader

Re: Auto loader

Re: Auto loader