topic Re: Roll back to previous version of an AutoLoader checkpoint file in Data Engineering

Roll back to previous version of an AutoLoader checkpoint file

DomDuf — Mon, 24 Apr 2023 20:41:04 GMT

I know to "reset" AutoLoader, you can delete the checkpoint file entirely. I was wondering if it's possible to and how would someone :

Get the checkpoint file to a previous version so I can reload certain files that were already processed
Delete certain rows in the checkpoint file (by creation date for example)

Thanks for your help.

Re: Roll back to previous version of an AutoLoader checkpoint file

Murthy1 — Tue, 25 Apr 2023 09:43:08 GMT

Hello @Dominic Dufour ,

I had the same question and Unfortunately it is not possible yet.

However you can delete the checkpoint file and use the Autoloader option - "modifiedAfter" to pick up the files after a specific time. This is a one time activity as the checkpoint will be created again to continue for the future loads. You can remove the "modifiedAfter" once your data loads are back on track.

The other work around I can think of is a bit intensive one, You can have some scheduled jobs to copy the checkpoint files to a different location frequently (with date time markers- probably in the folder name). You can restore the checkpoint files from that location when you want to reset Autoloader.

Re: Roll back to previous version of an AutoLoader checkpoint file

DomDuf — Tue, 25 Apr 2023 14:08:11 GMT

Thanks for the quick answer, this was really helpful !

Re: Roll back to previous version of an AutoLoader checkpoint file

MRTN — Tue, 16 May 2023 13:29:06 GMT

This would for sure be a useful feature.