I am trying to use Databricks Autoloader with File Notification Mode

[mk1987c]

When i run my command for readstream using  .option("cloudFiles.useNotifications", "true") it start reading the files from Azure blob (please note that i did not provide the configuration like subscription id , clint id , connect string and all while reading )

df = (

  spark.readStream.format("cloudFiles")

  .option("cloudFiles.format", "csv")

  .option("cloudFiles.useNotifications", "true")

  .option("header", True)

  .load(source_data_loc)

)

now when i start writing it using below code

df.writeStream.format("delta").option("checkpointLocation", checkpoints_loc).outputMode("append").start(target_data_loc)

it started giving me error like Please provide the subscription ID with `cloudFiles.subscriptionId`

Now to resolve these issue i gave all these info as mention below while using readstream

cloudFilesConf = {

 "cloudFiles.subscriptionId": subscriptionId,

 "cloudFiles.clientId": spn_client_id,

 "cloudFiles.connectionString": QueueSASKey,

 "cloudFiles.clientSecret": spn_secret_name,

 "cloudFiles.tenantId": spn_tenant_id,

 "cloudFiles.resourceGroup": ResourceGroup_name,  

 "cloudFiles.schemaLocation": schema_loc,

 #"cloudFiles.useNotifications": "true"

}

but when i am trying to run now it is giving me "option() got an unexpected keyword argument 'cloudFiles.subscriptionId'" this error so not sure where the issue is

Please suggest

[jose_gonzalez]

Hi,

I would like to share the following docs that might be able to help you with this issue. https://docs.databricks.com/ingestion/auto-loader/file-notification-mode.html#required-permissions-f... you need to set the right permission and define all the settings to be able to consume data.

[mk1987c · Accepted solution]

thanks you i found the issue and it is resolved now thanks

[Abhradwip]

Please let us know how the issue got resolved

[manish1987c]

you need to create event grid subscriptions and queues should already provisioned and it should be part of the CI/CD process

[mk1987c]

Hi Abhradwip, the issue is resolved.