topic Re: Autoloader Checkpoint Fails and then the after changing the checkpoint path need to reload all d in Data Engineering

Autoloader Checkpoint Fails and then the after changing the checkpoint path need to reload all data

Subhasis — Wed, 04 Sep 2024 05:31:32 GMT

Autoloader Checkpoint Fails and then the after changing the checkpoint path need to reload all data. I want to load all the data which are not processed . I don't want to relaod all the data.

Re: Autoloader Checkpoint Fails and then the after changing the checkpoint path need to reload all d

szymon_dybczak — Wed, 04 Sep 2024 05:53:13 GMT

Hi @Subhasis ,

Could you provide more details? Like exact error message, you autoloader configuration etc. It will be easier for us to help you

Re: Autoloader Checkpoint Fails and then the after changing the checkpoint path need to reload all d

daniel_sahal — Wed, 04 Sep 2024 05:54:43 GMT

@Subhasis
What do you exactly mean by "Autoloader Checkpoint fails"?
How did you change the checkpoint path? Simply by specifying a new path?
If yes, then it's normal that it will try to reload all data, as it sees that the checkpoint is empty thus it's trying to load everything it finds.
What you could do is to specify modifiedAfter option to set up a cutoff date.

Re: Autoloader Checkpoint Fails and then the after changing the checkpoint path need to reload all d

Subhasis — Wed, 04 Sep 2024 06:28:08 GMT

No such error is showing it is reading file but it is not writing the data into deltatable. Then when I identified it is not writing data I created a new checkpoint path. Then it is reloading all the data. How to avoid this situation . Modified after I used but then not able to identify from when the data is not writing since the job is not failing as such.

Re: Autoloader Checkpoint Fails and then the after changing the checkpoint path need to reload all d

Subhasis — Wed, 04 Sep 2024 06:30:24 GMT

Do checkpoint has some benchmark capacity after that it stops writing data?

Re: Autoloader Checkpoint Fails and then the after changing the checkpoint path need to reload all d

szymon_dybczak — Wed, 04 Sep 2024 06:36:53 GMT

You can use cloud_files_state function to see what files has been processed by autoloader and saved in checkpoint.

I'm assuming that in your case you have some misconfiguration that's causing a problem

cloud_files_state table-valued function | Databricks on AWS