Help Needed: Grants remaining for removed Groups and Service Principal

[JoeStringer]

We have an issue in a Workspace which is managed by Terraform, a change went in to update the Group and Service Principal (SP) names but due to the internal ordering the Groups and SP were removed and replaced before the Grants were updated.

If we now try to remove the Grants for the old Group/SP through Terraform, the UI or CLI, we cannot, and receive errors saying similar to: "the principal for the <entity> cannot be found" (where <entity> is a group name or SP ID)

The strangest thing though is when listing the grants, the groups that have been deleted in the Account/Workspace occasionally don't show, but then return after a few seconds. Repeatedly running the cli command to get the Grants on the affected catalogs show the groups returning and disappearing repeatedly, this can also be seen in the UI.

In the below outputs "Bronze group", "Silver group", "Gold group" and "unity_admins" are the groups that have been deleted, with "<old service principal id>" also being removed.

This first request shows all the groups:

✦ ❯ databricks grants get CATALOG production
{
"privilege_assignments": [
{
"principal":"Bronze group",
"privileges": [
"ALL_PRIVILEGES"
]
},
{
"principal":"Bronze",
"privileges": [
"ALL_PRIVILEGES"
]
},
{
"principal":"<old service principal id>",
"privileges": [
"ALL_PRIVILEGES"
]
},
{
"principal":"unity_admins",
"privileges": [
"ALL_PRIVILEGES"
]
},
{
"principal":"UnityAdmins",
"privileges": [
"ALL_PRIVILEGES"
]
},
{
"principal":"Silver group",
"privileges": [
"USE_CATALOG"
]
},
{
"principal":"<new service principal id>",
"privileges": [
"ALL_PRIVILEGES"
]
},
{
"principal":"gold group",
"privileges": [
"USE_CATALOG"
]
}
]
}

 This next one a few seconds later returns, but without "Bronze Group" or "Silver Group"

✦ ❯ databricks grants get CATALOG production
{
"privilege_assignments": [
{
"principal":"Bronze",
"privileges": [
"ALL_PRIVILEGES"
]
},
{
"principal":"<old service principal id>",
"privileges": [
"ALL_PRIVILEGES"
]
},
{
"principal":"unity_admins",
"privileges": [
"ALL_PRIVILEGES"
]
},
{
"principal":"UnityAdmins",
"privileges": [
"ALL_PRIVILEGES"
]
},
{
"principal":"<new service principal id>",
"privileges": [
"ALL_PRIVILEGES"
]
},
{
"principal":"gold group",
"privileges": [
"USE_CATALOG"
]
}
]
}

I've tried recreating groups on the same names, but I'm unable to attach them from the Account to the Workspace with the similar error ("Failed to add group: Principal with id <group id> does not exist"), this appears to be the same underlying issue that there is some form of internal state in the Workspace that is not able to clean up/remove the references to the old Groups and Service Principal.

(We're currently unable to raise a Databricks Support request for this at this time, we've been advised to raise it here and await the next available Office Hours)

Any support appreciated!

Joe

[YennickT]

We are experiencing a similar issue, except with the storage credential resource. We created some storage credentials using Terraform, but when trying to destroy them using Terraform they were ignored. So we decided to manually delete the storage credentials through the UI, which worked. Then we reprovisioned the storage credentials with the same name using Terraform, and now we experience the same inconsistent behaviour with the CLI when querying the grants of the storage credential. In about 75% of the calls it returns the expected groups having access to the storage location, and in the other 25% it returns an empty result.

Note that this is not a CLI-only issue. This also happens when executing a `SHOW GRANTS ON STORAGE CREDENTIAL my_storage_credential` command using SQL.