stream from azure credentials
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-18-2023 03:08 AM
I am trying to read stream from azure:
(spark.readStream
.format("cloudFiles")
.option('cloudFiles.clientId', CLIENT_ID)
.option('cloudFiles.clientSecret', CLIENT_SECRET)
.option('cloudFiles.tenantId', TENTANT_ID)
.option("header", "true")
.option("cloudFiles.format", "csv")
.option("cloudFiles.schemaLocation", CHECKPOINT_PATH)
.load(f"wasbs://{CONTAINER}@{ACCOUNT_NAME}.blob.core.windows.net/"+AZURE_PATH)
)
yet I get
Py4JJavaError: An error occurred while calling o9451.load.
: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Container <container> in account <account>.blob.core.windows.net not found, and we can't create it using anoynomous credentials, and no credentials found for them in the configuration.
I know the location exists and it seems it ignores the provided credentials. How can I set the credentials?
- Labels:
-
Credentials
-
Spark streaming
-
Stream
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-10-2023 07:53 AM
@Hanan Shteingart :
It looks like you're using the Azure Blob Storage connector for Spark to read data from Azure. The error message suggests that the credentials you provided are not being used by the connector.
To specify the credentials, you can set the following options in your code:
.option("cloudFiles.accountName", ACCOUNT_NAME)
.option("cloudFiles.accountKey", ACCOUNT_KEY)
Replace ACCOUNT_NAME and ACCOUNT_KEY with the name and key of your Azure storage account, respectively. You can find the account key in the Azure portal under the "Access keys" section of your storage account.
Note that providing the account key in your code is not recommended for security reasons. Instead, you should consider using Azure Key Vault or other secure key management solutions to store and retrieve your credentials.