cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

I am trying to use Databricks Autoloader with File Notification Mode

mk1987c
New Contributor III

When i run my command for readstream using  .option("cloudFiles.useNotifications", "true") it start reading the files from Azure blob (please note that i did not provide the configuration like subscription id , clint id , connect string and all while reading )

df = (

  spark.readStream.format("cloudFiles")

  .option("cloudFiles.format", "csv")

  .option("cloudFiles.useNotifications", "true")

  .option("header", True)

  .load(source_data_loc)

)

now when i start writing it using below code

df.writeStream.format("delta").option("checkpointLocation", checkpoints_loc).outputMode("append").start(target_data_loc)

it started giving me error like Please provide the subscription ID with `cloudFiles.subscriptionId`

Now to resolve these issue i gave all these info as mention below while using readstream

cloudFilesConf = {

 "cloudFiles.subscriptionId": subscriptionId,

 "cloudFiles.clientId": spn_client_id,

 "cloudFiles.connectionString": QueueSASKey,

 "cloudFiles.clientSecret": spn_secret_name,

 "cloudFiles.tenantId": spn_tenant_id,

 "cloudFiles.resourceGroup": ResourceGroup_name,  

 "cloudFiles.schemaLocation": schema_loc,

 #"cloudFiles.useNotifications": "true"

}

but when i am trying to run now it is giving me "option() got an unexpected keyword argument 'cloudFiles.subscriptionId'" this error so not sure where the issue is

Please suggest

1 ACCEPTED SOLUTION

Accepted Solutions

thanks you i found the issue and it is resolved now thanks

View solution in original post

4 REPLIES 4

jose_gonzalez
Moderator
Moderator

Hi,

I would like to share the following docs that might be able to help you with this issue. https://docs.databricks.com/ingestion/auto-loader/file-notification-mode.html#required-permissions-f... you need to set the right permission and define all the settings to be able to consume data.

thanks you i found the issue and it is resolved now thanks

Abhradwip
New Contributor II

Please let us know how the issue got resolved

mk1987c
New Contributor III

Hi Abhradwip, the issue is resolved.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.