So my files arrive at the cloud storage and I have configured an autoloader to read these files.
The files have a monotonically increasing id in their name.
How can I detect a gap and stop the DLT as soon as there is a gap?
eg.
Autoloader finds file1, ingests
Autoloader finds file2, ingests
Autoloader finds file3, ingests
Autoloader finds file5 -> file4 is missing: STOP
Is this possible using DLT? Or should I go for a streaming job?
