topic Re: DLT+AutoLoader: where is the schema and checkpoint hide? in Data Engineering

DLT+AutoLoader: where is the schema and checkpoint hide?

Daba — Sun, 10 Apr 2022 15:22:33 GMT

Hi,

I'm exploring the DLT with AutoLoader feature and wondering where are the schema and checkpoint hide? I want to wipe these two to reset/reinitialize the flow but unlike the "regular" AutoLoader the checkpoint and schema folder are not there.

Thanks in advance.

Re: DLT+AutoLoader: where is the schema and checkpoint hide?

Hubert-Dudek — Mon, 11 Apr 2022 19:23:33 GMT

@Alexander Plepler , There is a storage option in pipeline settings - A path to a DBFS directory for storing checkpoints and tables created by the pipeline.

Additionally, delta is registered in metastore, so the table schema is there.

Re: DLT+AutoLoader: where is the schema and checkpoint hide?

Daba — Mon, 18 Apr 2022 13:37:58 GMT

I will. Thanks.

Re: DLT+AutoLoader: where is the schema and checkpoint hide?

Daba — Tue, 26 Apr 2022 11:16:26 GMT

Thanks Fatma,

I still can't find the place where the generated schema stored, but the initial problem is resolved by "Full Refresh" feature of DLT's job. So, I guess I'm ok.

Thanks.