topic I am trying to use Databricks Autoloader with File Notification Mode in Data Engineering

I am trying to use Databricks Autoloader with File Notification Mode

mk1987c — Sun, 12 Feb 2023 04:21:13 GMT

When i run my command for readstream using .option("cloudFiles.useNotifications", "true") it start reading the files from Azure blob (please note that i did not provide the configuration like subscription id , clint id , connect string and all while reading )

df = (

spark.readStream.format("cloudFiles")

.option("cloudFiles.format", "csv")

.option("cloudFiles.useNotifications", "true")

.option("header", True)

.load(source_data_loc)

)

now when i start writing it using below code

df.writeStream.format("delta").option("checkpointLocation", checkpoints_loc).outputMode("append").start(target_data_loc)

it started giving me error like Please provide the subscription ID with `cloudFiles.subscriptionId`

Now to resolve these issue i gave all these info as mention below while using readstream

cloudFilesConf = {

"cloudFiles.subscriptionId": subscriptionId,

"cloudFiles.clientId": spn_client_id,

"cloudFiles.connectionString": QueueSASKey,

"cloudFiles.clientSecret": spn_secret_name,

"cloudFiles.tenantId": spn_tenant_id,

"cloudFiles.resourceGroup": ResourceGroup_name,

"cloudFiles.schemaLocation": schema_loc,

#"cloudFiles.useNotifications": "true"

}

but when i am trying to run now it is giving me "option() got an unexpected keyword argument 'cloudFiles.subscriptionId'" this error so not sure where the issue is

Please suggest

Re: I am trying to use Databricks Autoloader with File Notification Mode

jose_gonzalez — Wed, 22 Feb 2023 22:27:59 GMT

Hi,

I would like to share the following docs that might be able to help you with this issue. https://docs.databricks.com/ingestion/auto-loader/file-notification-mode.html#required-permissions-for-configuring-file-notification-for-adls-gen2-and-azure-blob-storage you need to set the right permission and define all the settings to be able to consume data.

Re: I am trying to use Databricks Autoloader with File Notification Mode

mk1987c — Thu, 23 Feb 2023 00:22:52 GMT

thanks you i found the issue and it is resolved now thanks

Re: I am trying to use Databricks Autoloader with File Notification Mode

Abhradwip — Fri, 17 Mar 2023 08:01:25 GMT

Please let us know how the issue got resolved

Re: I am trying to use Databricks Autoloader with File Notification Mode

mk1987c — Tue, 21 Mar 2023 06:08:04 GMT

Hi Abhradwip, the issue is resolved.

Re: I am trying to use Databricks Autoloader with File Notification Mode

manish1987c — Tue, 04 Jun 2024 09:23:57 GMT

you need to create event grid subscriptions and queues should already provisioned and it should be part of the CI/CD process