topic Re: Question about notifications and failed jobs in Data Engineering

Question about notifications and failed jobs

f1nesse13 — Tue, 17 Dec 2024 19:07:31 GMT

Hello, I had a question involving rerunning a job from a checkpoint using ‘Repair Run’. I have a job which failed and Im looking to rerun the stream from a checkpoint. My job uses notifications for file detection (cloudFiles.useNotifications). My question is will the messages in the queue be consumed if Im rerunning from that checkpoint.

Re: Question about notifications and failed jobs

Louis_Frolio — Tue, 17 Dec 2024 20:40:35 GMT

When rerunning your job from a checkpoint using Repair Run with cloudFiles.useNotifications, only unprocessed messages in the queue (representing new or failed-to-process files) will be consumed. Files or events already recorded in the checkpoint will not trigger reprocessing unless you reset or delete the checkpoint metadata.

Hope this helps.

Louis.