When i run my command for readstream using .option("cloudFiles.useNotifications", "true") it start reading the files from Azure blob (please note that i did not provide the configuration like subscription id , clint id , connect string and all while reading )
df = (
spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "csv")
.option("cloudFiles.useNotifications", "true")
.option("header", True)
.load(source_data_loc)
)
now when i start writing it using below code
df.writeStream.format("delta").option("checkpointLocation", checkpoints_loc).outputMode("append").start(target_data_loc)
it started giving me error like Please provide the subscription ID with `cloudFiles.subscriptionId`
Now to resolve these issue i gave all these info as mention below while using readstream
cloudFilesConf = {
"cloudFiles.subscriptionId": subscriptionId,
"cloudFiles.clientId": spn_client_id,
"cloudFiles.connectionString": QueueSASKey,
"cloudFiles.clientSecret": spn_secret_name,
"cloudFiles.tenantId": spn_tenant_id,
"cloudFiles.resourceGroup": ResourceGroup_name,
"cloudFiles.schemaLocation": schema_loc,
#"cloudFiles.useNotifications": "true"
}
but when i am trying to run now it is giving me "option() got an unexpected keyword argument 'cloudFiles.subscriptionId'" this error so not sure where the issue is
Please suggest