stream from azure credentials

[chanansh]

I am trying to read stream from azure:

(spark.readStream
 .format("cloudFiles")
 .option('cloudFiles.clientId', CLIENT_ID)
 .option('cloudFiles.clientSecret', CLIENT_SECRET)
 .option('cloudFiles.tenantId', TENTANT_ID)
 .option("header", "true")
 .option("cloudFiles.format", "csv")
 .option("cloudFiles.schemaLocation", CHECKPOINT_PATH)
 .load(f"wasbs://{CONTAINER}@{ACCOUNT_NAME}.blob.core.windows.net/"+AZURE_PATH)
)

yet I get

Py4JJavaError: An error occurred while calling o9451.load.
: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Container <container> in account <account>.blob.core.windows.net not found, and we can't create it using anoynomous credentials, and no credentials found for them in the configuration.

I know the location exists and it seems it ignores the provided credentials. How can I set the credentials?

[Anonymous]

@Hanan Shteingart​ :

It looks like you're using the Azure Blob Storage connector for Spark to read data from Azure. The error message suggests that the credentials you provided are not being used by the connector.

To specify the credentials, you can set the following options in your code:

.option("cloudFiles.accountName", ACCOUNT_NAME)
.option("cloudFiles.accountKey", ACCOUNT_KEY)

Replace ACCOUNT_NAME and ACCOUNT_KEY with the name and key of your Azure storage account, respectively. You can find the account key in the Azure portal under the "Access keys" section of your storage account.

Note that providing the account key in your code is not recommended for security reasons. Instead, you should consider using Azure Key Vault or other secure key management solutions to store and retrieve your credentials.