cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Autoloader - File Notification mode

Gilg
Contributor II

Hi All,

I have set up a DLT pipeline that is using Autoloader in a file notification mode.

Everything runs smoothly for the first time. However, it seems like the next micro-batch did not trigger as I can see some events coming in the queue.

Gilg_0-1710827649089.png


But if I look at SparkUI I do not see any active jobs for a while now.

Gilg_1-1710827662118.png

Not sure what is happening here.

Cheers,

Gil

 

1 REPLY 1

Hi @Kaniz_Fatma 

I did some digging on the messages that we are receiving.

 

By default, autoloader generates the Event Grid System Topic, Event Subscriptions and Storage Queue endpoint.

 

Gilg_0-1710897237186.png

 

Gilg_6-1710897546457.png

 

Looking at the queue endpoint it has a filter that is set automatically below.

 

Gilg_2-1710897252022.png

 

 

In our test we have removed this filter and see what messages we are getting.

 

We’ve noticed that messages that we are receiving in the Storage Queue only have a tag of “CreateFile”.

Gilg_7-1710897593599.png

 

But autoloader seems to be listening to different api tags according to this below.

 

Gilg_4-1710897271297.png

 

 

 

I think that it could be the reason why we do not get any Active jobs in SparkUI because Autoloader is looking to different api tags to process.

Not sure why this is happening.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group