topic Re: AutoLoader with Custom Queue in Data Engineering

AutoLoader with Custom Queue

Soma — Mon, 13 Dec 2021 15:15:09 GMT

Hi Everyone can someone help with creating custom queue for auto loader as

given here as default FlushwithClose event is not getting created when my data is uploaded to blob as given in azure DB docs

cloudFiles.queueName

The name of the Azure queue. If provided, the cloud files source directly consumes events from this queue instead of setting up its own Azure Event Grid and Queue Storage services. In that case, your

cloudFiles.connectionString

requires only read permissions on the queue.

Re: AutoLoader with Custom Queue

Hubert-Dudek — Mon, 13 Dec 2021 15:43:52 GMT

you need to setup notification service for blob/adls like here https://docs.databricks.com/spark/latest/structured-streaming/auto-loader-gen2.html#cloud-resource-management

setUpNotificationServices will return queue name which later can be used in autoloader

Re: AutoLoader with Custom Queue

Soma — Mon, 13 Dec 2021 17:49:17 GMT

thanks HubertDudek quick questions here.

1.if I create a queue based on particular event can I just point that queue to manager alone or do I have to add some custom message

as I see data in queue with flush event contains some kind of json