Alberto_Umana
Databricks Employee
Databricks Employee

Hi @citizenX7042,

Since the error indicates an issue with the configuration value for fs.azure.account.key

Can you test with the below code:

 

from pyspark.sql.functions import regexp_extract, input_file_name

 

# Set the storage account key

spark.conf.set("fs.azure.account.key.<your-storage-account-name>.dfs.core.windows.net", "<your-storage-account-key>")

 

# Define the file path

single_file = "abfss://external-sources@<your-storage-account-name>.dfs.core.windows.net/Bronze/Tribe_Report/20241210/visa-10079563/cards-11-15967860899208-10079563-20241210.xml"

 

# Load the single file

raw_df_single = (

    spark.read.format("com.databricks.spark.xml")  # XML format

    .option("rowTag", "Card")                     # Specify the row tag for parsing

    .load(single_file)                            # Load the single file

    .withColumn("@FileName", regexp_extract(input_file_name(), r"([^/]+)$", 1))  # Extract file name

)

 

# Show a preview of the data

raw_df_single.show()