Hi all,
Is it correct that Azure-Databricks only support to write data to Azure Data Lake Gen2 and does not support for Azure Storage Blob (StorageV2 - general purpose) ?
In my case, I can read the data from Azure Storage Blob (StorageV2 - general purpose v2) to Databricks, but when writing data back from Databricks to that Azure blob, it shows error.
Here is my code
# Define the path where you want to write the table
output_path = "wasbs://powerbiinfo@dlsupeducationdev.blob.core.windows.net/staging/test"
# Write the DataFrame to the specified path in JSON format
try:
df.write.mode("overwrite").json(output_path)
print("Write operation successful.")
except Exception as e:
print(f"Error writing to Azure Blob Storage: {e}")
And error:
Error writing to Azure Blob Storage: An error occurred while calling o451.json. : shaded.databricks.org.apache.hadoop.fs.azure.AzureException: java.lang.IllegalArgumentException: The String is not a valid Base64-encoded string. at shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:1217)...."
Any idea, please help!
Thanks
Mo
