topic Errors When Using R on Unity Catalog Clusters in Data Engineering

Errors When Using R on Unity Catalog Clusters

Kayla — Tue, 16 Apr 2024 18:12:09 GMT

We are running into errors when running workflows with multiple jobs using the same notebook/different parameters. They are reading from tables we still have in hive_metastore, there's no Unity Catalog tables or functionality referenced anywhere. We're getting this error: "m.databricks.unity.error.MissingCredentialScopeException: [UNITY_CREDENTIAL_SCOPE_MISSING_SCOPE] Missing Credential Scope. Failed to find Unity Credential Scope"

This happens consistently for 2 out of 3 notebooks, so every workflow run has 1 success and 2 failures. Running the same code on a cluster that is not Unity Catalog enabled runs fine, but we will need that capability going forward. This is on Databricks 13.3 LTS.

Re: Errors When Using R on Unity Catalog Clusters

Kayla — Wed, 17 Apr 2024 13:36:12 GMT

Hi @Retired_mod
Thanks for the quick response.
To clarify- no Unity Catalog asset of any sort is referenced or involved. There's no environment variables in play. We're not using Databricks-Connect, we're running a Databricks notebook via a workflow. We're not using any AWS services. We're on GCP, and also not using any GCP services.

We have discovered that this works on non-Unity Catalog clusters, or if we run the notebooks sequentially instead of in parallel.
We're not clear why running these notebooks in parallel on a Unity Catalog cluster is giving "credential scope" errors when there's no reason for any credential scope to be needed.

Re: Errors When Using R on Unity Catalog Clusters

mariusatkinson — Wed, 01 May 2024 16:04:54 GMT

Do you have row based access on the data you are trying to access?

Re: Errors When Using R on Unity Catalog Clusters

Kayla — Wed, 01 May 2024 16:06:58 GMT

We do not, no.

Re: Errors When Using R on Unity Catalog Clusters

mariusatkinson — Wed, 01 May 2024 16:12:11 GMT

Ah, I suspected that it might have something to do with fine grained access control and an incompatability with R and UC when it's configured like in that way. Obvisouly if you don't, it's not that.

Re: Errors When Using R on Unity Catalog Clusters

Anwarchubb — Wed, 26 Mar 2025 08:46:30 GMT

R enabled cluster only supports single user group so please check permission at your group level