DBR 15.4 LTS Beta Unable to Write Files to Azure Storage Account

Brian-Nowak — Mon, 05 Aug 2024 16:51:20 GMT

Hi there!

I believe I might have identified a bug with DBR 15.4 LTS Beta. The basic task of saving data to a delta table, as well as an even more basic operation of saving a file to cloud storage, is failing on 15.4, but working perfectly fine on 15.3. Some other details:

it seems like this is only happening on a single node, single user cluster (still has the Unity Catalog tag)
the Storage Credential to this entire storage account is configured using a system-assigned managed identity, per Databricks recommendation here
the access connector identity has been granted the following roles in the storage account (again, following the article): Storage Account Contributor, Storage Blob Data Contributor, Storage Queue Data Contributor
running the exact same command, on the exact same cluster fails on 15.4 but succeeds with 15.3
I'm running the notebook as my personal user, and I have ALL PRIVILEGES on this dev_sandbox catalog

the command - a simple saveAsTable:

table_path = "dev_sandbox.files_metadata.bntestsave" df.write.format("delta").mode("overwrite").saveAsTable(table_path)

Here is the error message, with some identifiable info redacted:

Failed to save file to /Volumes/[REDACTED]. Error: An error occurred while calling o1607.save. : org.apache.spark.SparkException: Exception thrown in awaitResult: Failed with shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException while processing file/directory :[[REDACTED]/_committed_4591969627371730351] in method:[Failed to acquire a SAS token for write on [REDACTED]/_committed_4591969627371730351 due to com.databricks.unity.error.MissingCredentialScopeException: [UNITY_CREDENTIAL_SCOPE_MISSING_SCOPE] Missing Credential Scope. Failed to find Unity Credential Scope.. SQLSTATE: XXKUC]

Re: DBR 15.4 LTS Beta Unable to Write Files to Azure Storage Account

wout__ — Wed, 07 Aug 2024 09:22:51 GMT

We have the same issue at 15.3 ML. It occurs when the output table is relatively large (or has multiple partitions - have not been able to validate). Using `.limit(200000)` worked in our case, but that might be because then there is only 1 partition.

Confirm that it works with 14.3 LTS

Re: DBR 15.4 LTS Beta Unable to Write Files to Azure Storage Account

Ricklen — Wed, 07 Aug 2024 09:55:16 GMT

We have the same issue since yesterday (6/8/2024), running on DBR 15.3 or 15.4 LTS Beta. It seems to have something to do with large table's indeed. Tried with multiple .partition sizes.

Re: DBR 15.4 LTS Beta Unable to Write Files to Azure Storage Account

Ricklen — Wed, 07 Aug 2024 11:03:14 GMT

Since I can't edit my previous post anymore, I can confirm it works with 14.3 LTS.

topic DBR 15.4 LTS Beta Unable to Write Files to Azure Storage Account in Data Engineering

DBR 15.4 LTS Beta Unable to Write Files to Azure Storage Account

Re: DBR 15.4 LTS Beta Unable to Write Files to Azure Storage Account

Re: DBR 15.4 LTS Beta Unable to Write Files to Azure Storage Account

Re: DBR 15.4 LTS Beta Unable to Write Files to Azure Storage Account