12-17-2024 06:07 AM - edited 12-17-2024 06:08 AM
I am currently utilizing a specific Service Principal in my DevOps steps to utilize the Databricks CLI. It's using the OAuth tokens with M2M authentication (Authenticate access to Azure Databricks with a service principal using OAuth (OAuth M2M) - Azure Dat...). And I use the Client ID and Client Secret, which I store (as plain text) to the datarbicks config file (.databrickscfg ) for authorization.
I created the secret manually and uploaded it to KeyVault. But I want to programatically renew this Secret every 2 months. Otherwise it is not in line with the security requirements of my company. However, I don't see a straightforward way of doing this through the Databricks CLI. Can someone help with this?
This question is also asked (amongst other questions) in this thread:
Security Consideration for OAUTH Secrets to use Se... - Databricks Community - 78227
2 weeks ago - last edited 2 weeks ago
After filing a Microsoft Support Ticket through my client they provided me with the solution to the inquiry. There seems to be a undocumented API call that you can do to create this SP Oauth Client Secret and it works perfectly:
curl -X POST --header "Authorization: Bearer <token>" https://<workspace-host-url>/api/2.0/accounts/servicePrincipals/<service-principal-id>/credentials/secrets
This is one of the open issues at Databricks (https://github.com/databricks/terraform-provider-databricks/issues/3363#issuecomment-2122177117)
12-17-2024 06:11 AM
Hi @Rob_Lemmens,
To programmatically renew OAuth secrets for a Service Principal every 2 months, you can follow these steps:
Example Azure Function to Renew OAuth Secret
import os
import requests
from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient
# Azure Key Vault details
key_vault_name = os.environ[
https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/oauth-m2m
12-17-2024 06:23 AM
Thanks @Alberto_Umana, for the quick reply
So I want to know about step 3. Could you elaborate on that?
12-17-2024 06:49 AM
Hi @Rob_Lemmens,
There is no direct method to renew an OAuth secret. Instead, you can create a new OAuth secret and replace the old one. You might need to create an Azure Function or Azure Automation runbook that will execute the renewal process every 2 months, but to replace the token, unfortunately cannot be renewed.
12-17-2024 07:14 AM - edited 12-17-2024 07:15 AM
Hi @Alberto_Umana,
That's perfectly fine, to replace it. For me replacing the old secret with a new secret is effectively the same as renewing the secret. So could you help me with how to replace the secret?
And if it is simply using the Databricks REST API. I imagine this should also be executable for agents in Azure Devops Pipeline instead of Azure Functions right?
12-18-2024 11:21 PM
Hi
This question is not answered yet. Could someone help me with it? Or is it not possible to programatically update oauth secrets through the Databricks REST API?
12-19-2024 06:48 AM
Hi @Rob_Lemmens - please refer to the API documentation:
https://docs.databricks.com/api/workspace/secrets/createscope
https://docs.databricks.com/api/workspace/secrets
curl -X POST https://<databricks-instance>/api/2.0/secrets/scopes/create \
-H "Authorization: Bearer <your-access-token>" \
-d '{
"scope": "my-scope",
"initial_manage_principal": "users"
}'
Replace <databricks-instance>
with your Databricks workspace URL and <your-access-token>
with your Databricks access token.
https://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/auth-with-azure-devops
12-20-2024 02:34 AM
Hi @Alberto_Umana,
Thanks for reaching out again.
The second point with the link for auth-with-azure-devops seems promising. I will look into it and let you know if it helped!
The first links to the API for the secrets are not the secrets I have been referring to in this thread. I was referring to the OAuth Secret of a Service Principal (I added a screenshot to the comment)
12-23-2024 08:56 AM
Hi @Rob_Lemmens,
Thanks for clarifying about the secrets. So this one might help you: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/oauth-m2m
2 weeks ago - last edited 2 weeks ago
After filing a Microsoft Support Ticket through my client they provided me with the solution to the inquiry. There seems to be a undocumented API call that you can do to create this SP Oauth Client Secret and it works perfectly:
curl -X POST --header "Authorization: Bearer <token>" https://<workspace-host-url>/api/2.0/accounts/servicePrincipals/<service-principal-id>/credentials/secrets
This is one of the open issues at Databricks (https://github.com/databricks/terraform-provider-databricks/issues/3363#issuecomment-2122177117)
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group