Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
β06-29-2022 01:08 PM
How to get a list of file loaded by auto loader?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
β06-29-2022 01:20 PM
If you are using the checkpointLocationoption you can read all the files that were processed by reading the rocksDB logs. Some example code to achieve that, note that you need to point to the path on the checkpoint location that you want to retrieve the loaded files list.
from glob import glob
import codecs
directory = "<YOUR_PATH_GOES_HERE>/sources/*/rocksdb/logs/"
for file in glob(f"{directory}/*.log"):
with codecs.open(file, encoding='utf-8', errors='ignore') as f:
f = f.readlines()
print(f)
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
β06-29-2023 12:21 PM
Itβs great feature and saves time