DLT+AutoLoader: where is the schema and checkpoint hide?

[Daba]

Hi, 

I'm exploring the DLT with AutoLoader feature and wondering where are the schema and checkpoint hide? I want to wipe these two to reset/reinitialize the flow but unlike the "regular" AutoLoader the checkpoint and schema folder are not there.

Thanks in advance.

[Hubert-Dudek]

@Alexander Plepler​ , There is a storage option in pipeline settings - A path to a DBFS directory for storing checkpoints and tables created by the pipeline.

Additionally, delta is registered in metastore, so the table schema is there.

[Kaniz]

Hi @Alexander Plepler​ , Please check the pipeline settings in this document.

[Daba]

I will. Thanks.

[Kaniz]

Hi @Alexander Plepler​ , Just a friendly follow-up. Do you still need help? Please let us know.

[Daba · Accepted solution]

Thanks Fatma,

I still can't find the place where the generated schema stored, but the initial problem is resolved by "Full Refresh" feature of DLT's job. So, I guess I'm ok.

Thanks.

[Kaniz]

Hi @Alexander Plepler​ , Thank you for the update. Please help the community by marking an answer as the best.