DBR 15.4 LTS Beta Unable to Write Files to Azure Storage Account

[Brian-Nowak]

Hi there!

I believe I might have identified a bug with DBR 15.4 LTS Beta. The basic task of saving data to a delta table, as well as an even more basic operation of saving a file to cloud storage, is failing on 15.4, but working perfectly fine on 15.3. Some other details:  

• it seems like this is only happening on a single node, single user cluster (still has the Unity Catalog tag)
• the Storage Credential to this entire storage account is configured using a system-assigned managed identity, per Databricks recommendation here
• the access connector identity has been granted the following roles in the storage account (again, following the article): Storage Account Contributor, Storage Blob Data Contributor, Storage Queue Data Contributor
• running the exact same command, on the exact same cluster fails on 15.4 but succeeds with 15.3
• I'm running the notebook as my personal user, and I have ALL PRIVILEGES on this dev_sandbox catalog

the command - a simple saveAsTable:

table_path = "dev_sandbox.files_metadata.bntestsave"
df.write.format("delta").mode("overwrite").saveAsTable(table_path)

Here is the error message, with some identifiable info redacted: 

Failed to save file to /Volumes/[REDACTED]. Error: An error occurred while calling o1607.save.
: org.apache.spark.SparkException: Exception thrown in awaitResult: Failed with shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException while processing file/directory :[[REDACTED]/_committed_4591969627371730351] in method:[Failed to acquire a SAS token for write on [REDACTED]/_committed_4591969627371730351 due to com.databricks.unity.error.MissingCredentialScopeException: [UNITY_CREDENTIAL_SCOPE_MISSING_SCOPE] Missing Credential Scope. Failed to find Unity Credential Scope.. SQLSTATE: XXKUC]

[wout__]

We have the same issue at 15.3 ML. It occurs when the output table is relatively large (or has multiple partitions -  have not been able to validate). Using `.limit(200000)` worked in our case, but that might be because then there is only 1 partition.

Confirm that it works with 14.3 LTS

[Ricklen]

We have the same issue since yesterday (6/8/2024), running on DBR 15.3 or 15.4 LTS Beta. It seems to have something to do with large table's indeed. Tried with multiple .partition sizes.

[Ricklen]

Since I can't edit my previous post anymore, I can confirm it works with 14.3 LTS.

[Kaniz_Fatma]

Hi @Brian-Nowak, first, verify that Unity Catalog is correctly configured and that credential scopes are properly set. Check the permissions for the system-assigned managed identity, as delays in permission propagation might be an issue. Try running the command on a different cluster configuration, and if the problem persists, consider using DBR 15.3 as a temporary workaround while you wait for a fix in 15.4. Make sure your Databricks environment is up-to-date with the latest patches, and report this as a bug to Databricks support for more specific guidance.