โ04-16-2024 11:12 AM
We are running into errors when running workflows with multiple jobs using the same notebook/different parameters. They are reading from tables we still have in hive_metastore, there's no Unity Catalog tables or functionality referenced anywhere. We're getting this error: "m.databricks.unity.error.MissingCredentialScopeException: [UNITY_CREDENTIAL_SCOPE_MISSING_SCOPE] Missing Credential Scope. Failed to find Unity Credential Scope"
This happens consistently for 2 out of 3 notebooks, so every workflow run has 1 success and 2 failures. Running the same code on a cluster that is not Unity Catalog enabled runs fine, but we will need that capability going forward. This is on Databricks 13.3 LTS.
โ04-17-2024 04:40 AM
Hi @Kayla,
Good luck with resolving the credential scope issue! ๐
โ04-17-2024 06:36 AM
Hi @Kaniz_Fatma
Thanks for the quick response.
To clarify- no Unity Catalog asset of any sort is referenced or involved. There's no environment variables in play. We're not using Databricks-Connect, we're running a Databricks notebook via a workflow. We're not using any AWS services. We're on GCP, and also not using any GCP services.
We have discovered that this works on non-Unity Catalog clusters, or if we run the notebooks sequentially instead of in parallel.
We're not clear why running these notebooks in parallel on a Unity Catalog cluster is giving "credential scope" errors when there's no reason for any credential scope to be needed.
โ05-01-2024 09:04 AM
Do you have row based access on the data you are trying to access?
โ05-01-2024 09:06 AM
We do not, no.
โ05-01-2024 09:12 AM
Ah, I suspected that it might have something to do with fine grained access control and an incompatability with R and UC when it's configured like in that way. Obvisouly if you don't, it's not that.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group