OAUTH Secrets Rotation for Service Principal through Databricks CLI

[Rob_Lemmens]

I am currently utilizing a specific Service Principal in my DevOps steps to utilize the Databricks CLI. It's using the OAuth tokens with M2M authentication (Authenticate access to Azure Databricks with a service principal using OAuth (OAuth M2M) - Azure Dat...). And I use the Client ID and Client Secret, which I store (as plain text) to the datarbicks config file (.databrickscfg ) for authorization.

I created the secret manually and uploaded it to KeyVault. But I want to programatically renew this Secret every 2 months. Otherwise it is not in line with the security requirements of my company. However, I don't see a straightforward way of doing this through the Databricks CLI. Can someone help with this?

This question is also asked (amongst other questions) in this thread:
Security Consideration for OAUTH Secrets to use Se... - Databricks Community - 78227  

[Alberto_Umana]

Hi @Rob_Lemmens,

To programmatically renew OAuth secrets for a Service Principal every 2 months, you can follow these steps:

1. Create a Service Principal and OAuth Secret:
• Follow the steps in the Azure Databricks documentation to create a Service Principal and generate an OAuth secret.
2. Store the Secret in Azure Key Vault:
• Store the generated OAuth secret in Azure Key Vault for secure access.
3. Automate Secret Renewal:
• Use Azure Functions or Azure Automation to create a scheduled task that runs every 2 months.
• The task should:
• Generate a new OAuth secret using the Databricks REST API.
• Update the secret in Azure Key Vault.
4. Update .databrickscfg File:
• Ensure your DevOps pipeline retrieves the latest secret from Azure Key Vault and updates the .databrickscfg file before using the Databricks CLI.

Example Azure Function to Renew OAuth Secret

import os

import requests

from azure.identity import DefaultAzureCredential

from azure.keyvault.secrets import SecretClient

# Azure Key Vault details

key_vault_name = os.environ[

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/oauth-m2m

[Rob_Lemmens]

Thanks @Alberto_Umana, for the quick reply

So I want to know about step 3. Could you elaborate on that?

• Because so far I only see ways to create actual OAuth tokens in Databricks REST API. I don't see how to actually renew the OAuth secret.

[Alberto_Umana]

Hi @Rob_Lemmens,

There is no direct method to renew an OAuth secret. Instead, you can create a new OAuth secret and replace the old one. You might need to create an Azure Function or Azure Automation runbook that will execute the renewal process every 2 months, but to replace the token, unfortunately cannot be renewed.

[Rob_Lemmens]

Hi @Alberto_Umana,

That's perfectly fine, to replace it. For me replacing the old secret with a new secret is effectively the same as renewing the secret. So could you help me with how to replace the secret?

And if it is simply using the Databricks REST API. I imagine this should also be executable for agents in Azure Devops Pipeline instead of Azure Functions right?

[Rob_Lemmens]

Hi

This question is not answered yet. Could someone help me with it? Or is it not possible to programatically update oauth secrets through the Databricks REST API?

[Alberto_Umana]

Hi @Rob_Lemmens - please refer to the API documentation:

https://docs.databricks.com/api/workspace/secrets/createscope

https://docs.databricks.com/api/workspace/secrets

curl -X POST https://<databricks-instance>/api/2.0/secrets/scopes/create \
-H "Authorization: Bearer <your-access-token>" \
-d '{
"scope": "my-scope",
"initial_manage_principal": "users"
}'

Replace <databricks-instance> with your Databricks workspace URL and <your-access-token> with your Databricks access token.

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/auth-with-azure-devops