cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

AutoLoader with Custom Queue

Soma
Valued Contributor

Hi Everyone can someone help with creating custom queue for auto loader as

given here as default FlushwithClose event is not getting created when my data is uploaded to blob as given in azure DB docs

cloudFiles.queueName

The name of the Azure queue. If provided, the cloud files source directly consumes events from this queue instead of setting up its own Azure Event Grid and Queue Storage services. In that case, your 

cloudFiles.connectionString

 requires only read permissions on the queue.

1 ACCEPTED SOLUTION

Accepted Solutions

Hubert-Dudek
Esteemed Contributor III

you need to setup notification service for blob/adls like here https://docs.databricks.com/spark/latest/structured-streaming/auto-loader-gen2.html#cloud-resource-m...

setUpNotificationServices will return queue name which later can be used in autoloader

View solution in original post

3 REPLIES 3

Hubert-Dudek
Esteemed Contributor III

you need to setup notification service for blob/adls like here https://docs.databricks.com/spark/latest/structured-streaming/auto-loader-gen2.html#cloud-resource-m...

setUpNotificationServices will return queue name which later can be used in autoloader

Soma
Valued Contributor

thanks HubertDudek quick questions here.

1.if I create a queue based on particular event can I just point that queue to manager alone or do I have to add some custom message

as I see data in queue with flush event contains some kind of json

Kaniz
Community Manager
Community Manager

Hi @somanath Sankaran​ , Please go through the practical example here.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.