szymon_dybczak
Esteemed Contributor III

Hi @ibrahim21124 ,

If you are using auto loader you also implicitly using RockDB - it stores state data at a checkpoint position within the RocksDB key-value store.
To find your checkpoint location follow the answer of Jacek Laskowski: https://stackoverflow.com/a/75716559
Also, under the hood autoloader is based on spark structured streaming model. I think it will be good for you to familiarize with documention, definitly will clarify a lot of concepts like checkpointing, state store etc.

Structured Streaming Programming Guide - Spark 3.2.0 Documentation (apache.org)