Autoloader - File Notification mode

Gilg — Tue, 19 Mar 2024 05:56:39 GMT

Hi All,

I have set up a DLT pipeline that is using Autoloader in a file notification mode.

Everything runs smoothly for the first time. However, it seems like the next micro-batch did not trigger as I can see some events coming in the queue.

But if I look at SparkUI I do not see any active jobs for a while now.

Not sure what is happening here.

Cheers,

Gil

Re: Autoloader - File Notification mode

Gilg — Wed, 20 Mar 2024 01:20:22 GMT

Hi @Retired_mod

I did some digging on the messages that we are receiving.

By default, autoloader generates the Event Grid System Topic, Event Subscriptions and Storage Queue endpoint.

Looking at the queue endpoint it has a filter that is set automatically below.

In our test we have removed this filter and see what messages we are getting.

We’ve noticed that messages that we are receiving in the Storage Queue only have a tag of “CreateFile”.

But autoloader seems to be listening to different api tags according to this below.

I think that it could be the reason why we do not get any Active jobs in SparkUI because Autoloader is looking to different api tags to process.

Not sure why this is happening.

topic Autoloader - File Notification mode in Data Engineering

Autoloader - File Notification mode

Re: Autoloader - File Notification mode