topic Re: Unity catalog - Service Principal SCIM API account unauthorized in Data Governance

Unity catalog - Service Principal SCIM API account unauthorized

yvuignie — Fri, 09 Sep 2022 15:29:18 GMT

Hi,

Is it possible to create groups at the account level in Unity Catalog as a Service Principal ?

I can manage to create groups when authenticated as a user, but not as a Service Principal. I then get an error "user not authorized".

The service principal has the role Account admin visible in the account console and can create other workspace's resources related, as well as metastore using the terraform provider with the host provided as the url of a workspace (but can't manage to use the provider with host https://accounts.azuredatabricks.net, kind of similar issue as https://community.databricks.com/s/question/0D58Y000098lPUkSAM/uc-service-principalterraform).

I tried with terraform as well as Postman via SCIM API 2.0 (Accounts) ({{baseUrl}}/accounts/:account_id/scim/v2/Groups) using the token generated with "az account get-access-token"

The error with terraform:

"Error: cannot create group: User not authorized. Using azure-client-secret auth: host=https://accounts.azuredatabricks.net, account_id=..."

I've read the documentation here: https://docs.microsoft.com/en-us/azure/databricks/administration-guide/users-groups/groups, but haven't found anything related to a service principal restriction.

Thanks for your help

Re: Unity catalog - Service Principal SCIM API account unauthorized

User16741082858 — Sun, 11 Sep 2022 05:18:55 GMT

Hi @Yannick Vuignier!

What is the user attribute/role stored in AAD? Make sure that the service principal is assigned the Contributor or Owner role in your Azure portal

Re: Unity catalog - Service Principal SCIM API account unauthorized

yvuignie — Tue, 13 Sep 2022 08:51:56 GMT

Hi @Pearl Ubaru ,

Thank you for your answer.

The service principal has the role Owner of the subscription.

Re: Unity catalog - Service Principal SCIM API account unauthorized

User16741082858 — Tue, 13 Sep 2022 15:16:23 GMT

Okay, no problem. So you cannot authenticate into the accounts console yet. We will soon preview Oauth tokens but not sure when. You can add service principals and give them account admin rights by using SCIM tokens. You can also add groups as well via SCIM, as long as you are the account owner or account admin. Here is a document that might be helpful - https://docs.databricks.com/administration-guide/users-groups/service-principals.html#assign-account-admin-rights-to-a-service-principal

Re: Unity catalog - Service Principal SCIM API account unauthorized

yvuignie — Tue, 13 Sep 2022 15:31:28 GMT

Thank you for your help. But the service principal is indeed "Account admin", the tag appears in the account console, the tab is on. Actually we want to use terraform to create groups using this service principal, but as it doesn't work, we tried directly with the API and we get the same result, "User not authorized".

Re: Unity catalog - Service Principal SCIM API account unauthorized

User16741082858 — Tue, 13 Sep 2022 16:10:13 GMT

Of course! Are you using identity federation?

Re: Unity catalog - Service Principal SCIM API account unauthorized

yvuignie — Wed, 14 Sep 2022 08:21:57 GMT

Yes, we have identity federation between the account and the workspaces.

We also have user provisioning enabled. We can manage to create groups with the SCIM token generated from the account console. If user provisioning is enabled, does this means that it is then required to use the SCIM token generated from the account console and that we can't use a service principal to manage groups?

Edit: Actually, it shouldn't be the case since I can create groups with my user using the SCIM API.

Re: Unity catalog - Service Principal SCIM API account unauthorized

Anonymous — Sat, 24 Sep 2022 08:18:55 GMT

Hey @Yannick Vuignier

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: Unity catalog - Service Principal SCIM API account unauthorized

yvuignie — Tue, 27 Sep 2022 07:13:46 GMT

Hi @Vidula Khanna ,

Thank you for your message, but no I am still not able to create groups using a service principal, same as before.

Re: Unity catalog - Service Principal SCIM API account unauthorized

User16741082858 — Tue, 27 Sep 2022 13:28:28 GMT

Hi @Yannick Vuignier,

You can use the permission assignment APIs - use PATs backed by SPs (https://api-docs.databricks.com/rest/latest/permission-assignment-account-api.html)

Re: Unity catalog - Service Principal SCIM API account unauthorized

yvuignie — Tue, 27 Sep 2022 15:07:00 GMT

Hello,

Thank you but I'm sorry I don't see how this API can help adding groups at the account level. Could you maybe please explain a bit ?

Re: Unity catalog - Service Principal SCIM API account unauthorized

User16741082858 — Tue, 27 Sep 2022 17:48:56 GMT

Hi! the issue is that you cannot create groups at the account level, right? You tried creating with Terraform but had an auth error. You tried with the API and still got an error code.

So the document I shared is for you is to authorize the service principal through the permissions API. I did find out however that it is not possible to do headless auth to the accounts console.

Please let me know if this makes sense!

Re: Unity catalog - Service Principal SCIM API account unauthorized

yvuignie — Wed, 28 Sep 2022 12:34:11 GMT

Thank you again for your answer! Yes you understand the issue well, terraform and the api is working with a user but not with a service principal.

But the permission assignment account API is unfortunately workspace related, all endpoints ask for a workspace_id, for instance this description says "Create or update workspace permissions for a principal". What is strange is that the service principal has the role "Account admin".

Re: Unity catalog - Service Principal SCIM API account unauthorized

Anonymous — Thu, 29 Sep 2022 06:58:00 GMT

Hi @Yannick Vuignier

Sorry for the inconvenience. I will forward your query to the respective person.

Thanks for your patience.

Re: Unity catalog - Service Principal SCIM API account unauthorized

Thilo — Fri, 30 Sep 2022 11:54:14 GMT

We do see the same problem. Any chance that headless authentication into the account will be made possible soon? Otherwise, it does not make sense to have "Account Admin" service principals.

Re: Unity catalog - Service Principal SCIM API account unauthorized

Dusko — Thu, 20 Oct 2022 06:44:52 GMT

Hello, any progress? Dealing with the same problem right now. Thanks

Re: Unity catalog - Service Principal SCIM API account unauthorized

yvuignie — Fri, 21 Oct 2022 15:01:53 GMT

I don't know what has been fixed in Databricks, but today it's finally working without any changes on our side.

Re: Unity catalog - Service Principal SCIM API account unauthorized

User16741082858 — Fri, 21 Oct 2022 15:27:29 GMT

Hi @Yannick Vuignier ! remember I let you know that the OAuth tokens were to preview soon? Well today, we enabled Azure AD token support for Service principals with Azure Databricks. So this means that you no longer need to use user principal tokens for API Automation with Azure DB.

Re: Unity catalog - Service Principal SCIM API account unauthorized

yvuignie — Tue, 25 Oct 2022 12:17:31 GMT

@Pearl Ubaru Thank you for your help