API - Service Principle Secret Generation

[yustus]

Hi everyone,

I am looking for a way to automatically create service principals, including their secrets, for M2M OAuth.

I know that service principals can be created via the API, but it seems that creating secrets for those service principals via the API is currently not supported.

Does anyone have an idea or workaround for this? Alternatively, do you know whether the Databricks engineers are already working on supporting this?

I know that OBO-tokens are an option, but security wise not a great choise.

Thank you in advance.

Kind regards

[szymon_dybczak · Accepted solution]

Hi @yustus ,

The AccountClient in the Databricks Python SDK exposes service_principal_secrets, which lets administrators create and manage OAuth secrets for service principals. The generated secrets can then be used to obtain OAuth access tokens for accessing both Databricks Account and Workspace APIs.

from databricks.sdk import AccountClient

a = AccountClient(
    host="https://accounts.azuredatabricks.net",
    account_id="<your-account-id>",
    client_id="<admin-sp-client-id>",
    client_secret="<admin-sp-secret>"
)

# Create SP
sp = a.service_principals.create(display_name="my-automation-sp")

# Create OAuth secret for it
secret = a.service_principal_secrets.create(service_principal_id=str(sp.id))

print(f"Client ID: {sp.application_id}")
print(f"Secret: {secret.secret}")  # Store this securely — shown only once!

You can also use terraform:

resource "databricks_service_principal" "sp" {
  provider     = databricks.account
  display_name = "my-automation-sp"
}

resource "databricks_service_principal_secret" "sp_secret" {
  service_principal_id = databricks_service_principal.sp.id
  lifetime             = "15552000s" # 180 days
}

If my answer was helpful, please consider marking it as accepted solution.

[ShamenParis]

Hi @yustus 

It’s great that you are prioritizing secure M2M OAuth over OBO tokens!

while @szymon_dybczak already mentioned the Python SDK and Terraform, I wanted to offer a few alternative approaches depending on how your automation is set up:

1. The Direct REST API (It is actually supported!) You mentioned that creating secrets via the API seems unsupported. The good news is that it actually is supported via the Account API (not the Workspace API). You can achieve this using a simple standard HTTP POST request. This is great if you don't want to rely on SDKs or Terraform. (Reference : https://docs.databricks.com/api/azure/account/accountserviceprincipals/create)

# Using standard cURL to the Account API
curl -X POST -H "Authorization: Bearer <your-admin-token>" \
https://accounts.azuredatabricks.net/api/2.0/accounts/<your-account-id>/servicePrincipals/<sp-id>/credentials/secrets

(Note: Be sure to replace the Azure URL with the AWS/GCP equivalent if you are on a different cloud).

2. The Databricks CLI (Best for simple bash/shell automation) If you are writing shell scripts for CI/CD, the new Databricks CLI natively supports secret generation and is much lighter than setting up a Python environment.

(Reference: https://docs.databricks.com/aws/en/dev-tools/cli/reference/service-principals-commands)

# Generate the secret directly via CLI
databricks service-principal-secrets create <service_principal_id>

3. Cloud-Native Identities (Zero Databricks Secrets) If you are using Azure, AWS, or GCP, the most secure workaround is to not generate Databricks secrets at all.

Instead of creating a Databricks-native OAuth secret, you can federate authentication to your cloud provider. For example, if you are on Azure, you can use an Azure AD Service Principal or a Managed Identity. You authenticate to Azure AD to get an Entra ID token, and Databricks will natively accept that token for M2M API calls.

This is often considered the gold standard for security because:

• You don't have to generate, rotate, or store Databricks secrets.

• Everything is managed centrally via your cloud provider's IAM.

Hope this gives you a few extra angles to tackle your automation!