Hello,

We have encountered a weird issue in our (old) set-up that looks like a bug in the Unity Catalog. The storage account which we are trying to persist is configured via External Volumes.

We have a pipeline that gets XML data and stores it in an RDD. The code then attempts to save the RDD, which causes the error: Invalid configuration value detected for fs.azure.account.key.

The weird thing is that this only happens when attempting to persist the RDD, but when attempting to persist a DataFrame, there is no issue.

path = 'abfss://......'
df = dummyDataframe()
# rdd_processed.collect() -> ['xml content 1', 'xml content 2', ..., 'xml content n']

df.write.text(path+'/staging1/') # works like a charm, can view saved files in SA
rdd_processed.saveAsTextFile(path+'/staging2/') # returns the error