How to detect gap in filenames (Autoloader)

L1000 — Thu, 19 Sep 2024 13:55:14 GMT

So my files arrive at the cloud storage and I have configured an autoloader to read these files.

The files have a monotonically increasing id in their name.

How can I detect a gap and stop the DLT as soon as there is a gap?

eg.

Autoloader finds file1, ingests

Autoloader finds file2, ingests

Autoloader finds file3, ingests

Autoloader finds file5 -> file4 is missing: STOP

Is this possible using DLT? Or should I go for a streaming job?

Re: How to detect gap in filenames (Autoloader)

SparkJun — Wed, 13 Nov 2024 06:43:17 GMT

It doesn't seem like this can be done through the DLT autoloader. Particularly you require an automatic stop without manual intervention. You can write a custom Structured Streaming job and use a sequence-checking logic, and foreachBatch to process incoming files and detect missing IDs.

topic Re: How to detect gap in filenames (Autoloader) in Data Engineering

How to detect gap in filenames (Autoloader)

Re: How to detect gap in filenames (Autoloader)