cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

DBR 15.4 LTS Beta Unable to Write Files to Azure Storage Account

Brian-Nowak
New Contributor II

Hi there!

I believe I might have identified a bug with DBR 15.4 LTS Beta. The basic task of saving data to a delta table, as well as an even more basic operation of saving a file to cloud storage, is failing on 15.4, but working perfectly fine on 15.3. Some other details:  

  • it seems like this is only happening on a single node, single user cluster (still has the Unity Catalog tag)
  • the Storage Credential to this entire storage account is configured using a system-assigned managed identity, per Databricks recommendation here
  • the access connector identity has been granted the following roles in the storage account (again, following the article): Storage Account Contributor, Storage Blob Data Contributor, Storage Queue Data Contributor
  • running the exact same command, on the exact same cluster fails on 15.4 but succeeds with 15.3
  • I'm running the notebook as my personal user, and I have ALL PRIVILEGES on this dev_sandbox catalog

the command - a simple saveAsTable:

 

table_path = "dev_sandbox.files_metadata.bntestsave"
df.write.format("delta").mode("overwrite").saveAsTable(table_path)

 

Here is the error message, with some identifiable info redacted: 

 

Failed to save file to /Volumes/[REDACTED]. Error: An error occurred while calling o1607.save.
: org.apache.spark.SparkException: Exception thrown in awaitResult: Failed with shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException while processing file/directory :[[REDACTED]/_committed_4591969627371730351] in method:[Failed to acquire a SAS token for write on [REDACTED]/_committed_4591969627371730351 due to com.databricks.unity.error.MissingCredentialScopeException: [UNITY_CREDENTIAL_SCOPE_MISSING_SCOPE] Missing Credential Scope. Failed to find Unity Credential Scope.. SQLSTATE: XXKUC]

 

3 REPLIES 3

wout__
New Contributor II

We have the same issue at 15.3 ML. It occurs when the output table is relatively large (or has multiple partitions -  have not been able to validate). Using `.limit(200000)` worked in our case, but that might be because then there is only 1 partition.

Confirm that it works with 14.3 LTS

Ricklen
New Contributor III

We have the same issue since yesterday (6/8/2024), running on DBR 15.3 or 15.4 LTS Beta. It seems to have something to do with large table's indeed. Tried with multiple .partition sizes.

Ricklen
New Contributor III

Since I can't edit my previous post anymore, I can confirm it works with 14.3 LTS.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group