Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-07-2024 08:58 AM
Hello!
I'm new on Databricks and I'm exploring some of its features.
I've successfully configured a workspace with unity catalog, one external storage location (ADLSg2) and the associated storage credential. I provided all privileges for all account users and try 'test connection' to ensure that everything is ok.
When I run the following command:
input_file_path = f"abfss://<my_container>@<my_storage_account>.dfs.core.windows.net<my_path>"
schema = spark.read.parquet(input_file_path).schema
I was able to read my parquet file and obtain the schema.
When I tried the following code to test the autoloader capabilities:
df = spark.readStream.format("cloudFiles") \
.option("cloudFiles.format", "parquet") \
.option("cloudFiles.inferColumnTypes", "true") \
.option("cloudFiles.schemaLocation", <schema_location_path>) \
.load(input_file_path)
I received the following error:
Failure to initialize configuration for storage account <my_storage_account>.dfs.core.windows.net: Invalid configuration value detected for fs.azure.account.keyInvalid...
How is it possibile that I can read my file using the standard read() function but I'm not able to read it with load()?
