When i run my command for readstream using  .option("cloudFiles.useNotifications", "true") it start reading the files from Azure blob (please note that i did not provide the configuration like subscription id , clint id , connect string and all while reading )

df = (

  spark.readStream.format("cloudFiles")

  .option("cloudFiles.format", "csv")

  .option("cloudFiles.useNotifications", "true")

  .option("header", True)

  .load(source_data_loc)

)

now when i start writing it using below code

df.writeStream.format("delta").option("checkpointLocation", checkpoints_loc).outputMode("append").start(target_data_loc)

it started giving me error like Please provide the subscription ID with `cloudFiles.subscriptionId`

Now to resolve these issue i gave all these info as mention below while using readstream

cloudFilesConf = {

 "cloudFiles.subscriptionId": subscriptionId,

 "cloudFiles.clientId": spn_client_id,

 "cloudFiles.connectionString": QueueSASKey,

 "cloudFiles.clientSecret": spn_secret_name,

 "cloudFiles.tenantId": spn_tenant_id,

 "cloudFiles.resourceGroup": ResourceGroup_name,  

 "cloudFiles.schemaLocation": schema_loc,

 #"cloudFiles.useNotifications": "true"

}

but when i am trying to run now it is giving me "option() got an unexpected keyword argument 'cloudFiles.subscriptionId'" this error so not sure where the issue is

Please suggest