Error with Read XML data using the spark-xml library
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-20-2025 02:48 AM
hi, would appritiate any help with an error with loading an XML file with spark-xml library.
my enviorment :
14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12)
library : com.databricks:spark-xml_2.12:0.15.0
on databricks notebook.
when running this script :
from pyspark.sql.functions import regexp_extract, input_file_name
print(single_file)
# Load the single file
raw_df_single = (
spark.read.format("com.databricks.spark.xml") # XML format
.option("rowTag", "Card") # Specify the row tag for parsing
.load(single_file) # Load the single file
.withColumn("@FileName", regexp_extract(input_file_name(), r"([^/]+)$", 1)) # Extract file name
)
# Show a preview of the data
raw_df_single.show()
i get an error :
Py4JJavaError: An error occurred while calling o621.load. : Failure to initialize configuration for storage account [REDACTED].dfs.core.windows.net: Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key
the print for single_file : abfss://external-sources@[REDACTED].dfs.core.windows.net/***/***/*/testfile.xml
it was tested and there is a file like that in the blob.
can library connect directly to the blob?
what is the format for that and the best practice?
it was tested and there is a file like that in the blob.
can library connect directly to the blob?
what is the format for that and the best practice?
