Unity Catalog - Invalid configuration value detected for fs.azure.account.key

[Kristjan]

Hi there,

I am having issue with writing a df to a table or display it. I have three dataframes that I have unioned and after I have done the union I cannot display the dataframe.

df_table1 = spark.sql(f'SELECT * FROM {sql_full_name}')
df_table2 = ...
df_table3 = ...
df_unioned = df_table1 .union(df_table2.union(df_table3 ))

display(df_table1)   // works fine
display(df_table2)   // works fine
display(df_unioned ) // ERROR

Failure to initialize configuration for storage account XXX.dfs.core.windows.net: Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key

I have manged to get it displayed, but for most parts it fails. I have configured the connection using Unity Catalog and External Locations, this has not been any issue until now.

Regards,
Kristjan

[Kristjan]

Thanks for your answer Kaniz,

It seems like this is just very misleading error message, by filtering these three tables, going from +30m rows down to 15m rows, everything works fine, except that I do not have all my rows in the table. In this case I have aggregated dates before 2019, otherwise I guess I would have to load the table without doing the union, or in batches.

Kristjan