Errors When Using R on Unity Catalog Clusters
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-16-2024 11:12 AM
We are running into errors when running workflows with multiple jobs using the same notebook/different parameters. They are reading from tables we still have in hive_metastore, there's no Unity Catalog tables or functionality referenced anywhere. We're getting this error: "m.databricks.unity.error.MissingCredentialScopeException: [UNITY_CREDENTIAL_SCOPE_MISSING_SCOPE] Missing Credential Scope. Failed to find Unity Credential Scope"
This happens consistently for 2 out of 3 notebooks, so every workflow run has 1 success and 2 failures. Running the same code on a cluster that is not Unity Catalog enabled runs fine, but we will need that capability going forward. This is on Databricks 13.3 LTS.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-17-2024 06:36 AM
Hi @Retired_mod
Thanks for the quick response.
To clarify- no Unity Catalog asset of any sort is referenced or involved. There's no environment variables in play. We're not using Databricks-Connect, we're running a Databricks notebook via a workflow. We're not using any AWS services. We're on GCP, and also not using any GCP services.
We have discovered that this works on non-Unity Catalog clusters, or if we run the notebooks sequentially instead of in parallel.
We're not clear why running these notebooks in parallel on a Unity Catalog cluster is giving "credential scope" errors when there's no reason for any credential scope to be needed.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-01-2024 09:04 AM
Do you have row based access on the data you are trying to access?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-01-2024 09:06 AM
We do not, no.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-01-2024 09:12 AM
Ah, I suspected that it might have something to do with fine grained access control and an incompatability with R and UC when it's configured like in that way. Obvisouly if you don't, it's not that.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
4 weeks ago
R enabled cluster only supports single user group so please check permission at your group level

