Hi there,
I am having issue with writing a df to a table or display it. I have three dataframes that I have unioned and after I have done the union I cannot display the dataframe.
df_table1 = spark.sql(f'SELECT * FROM {sql_full_name}')
df_table2 = ...
df_table3 = ...
df_unioned = df_table1 .union(df_table2.union(df_table3 ))
display(df_table1) // works fine
display(df_table2) // works fine
display(df_unioned ) // ERROR
Failure to initialize configuration for storage account XXX.dfs.core.windows.net: Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key
I have manged to get it displayed, but for most parts it fails. I have configured the connection using Unity Catalog and External Locations, this has not been any issue until now.
Regards,
Kristjan