Security Consideration for OAUTH Secrets to use Service Principal to authenticate with Databricks

[VJ3]

What are the security consideration we need to keep in mind when we want to us OAUTH Secrets to use a Service Principal to access Azure Databricks when Identity federation is disabled and workspace is not yet on boarded on to Unity Catalog?

Can we consider OAUTH secret similar to Personal Access Token?

What is time limit when OAUTH secrets expires?

How do we get new OAUTH secrets?

Can we use Azure Key Vault to store the OAUTH secrets?

What is the workflow we use in OAUTH for authentication? Do we use Implicit grant workflow in OAUTH?

Do we store secret in .databrickscfg?

Who has access to .databrickscfg?

How do we ensure that OAUTH secret is stored safely and encrypted using AES256 and higher encryption?

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/oauth-m2m

Regards,

VJ

[Kaniz_Fatma]

Hi @VJ3, 

OAuth Secrets vs Personal Access Tokens

OAuth secrets and personal access tokens are similar in that they both serve as authentication credentials for service principals. However, OAuth secrets are used specifically for OAuth machine-to-machine (M2M) authentication, while personal access tokens are used for workspace-level API access.

OAuth Secret Expiration and Rotation

OAuth secrets can expire after a configurable lifetime, up to a maximum of 1 year. It's recommended to rotate secrets regularly, such as every 30-90 days, to limit exposure if a secret is compromised. To generate a new OAuth secret, go to the service principal's settings page in the Azure Databricks workspace and click "Generate secret". The secret will only be displayed once during creation, so securely store it.

Secure Storage of OAuth Secrets

It's critical to store OAuth secrets securely. Azure Key Vault is an excellent option to encrypt and manage secrets. Alternatively, environment variables or a secure configuration file can be used. Never store OAuth secrets in plaintext in source code or configuration files like .databrickscfg. Only authorized users should have access to the secure storage location.

OAuth Authentication Workflow

For OAuth M2M authentication, the service principal uses the OAuth 2.0 client credentials flow. The service principal authenticates by presenting its client ID and client secret (OAuth secret) to the token endpoint. This differs from the implicit grant flow used for OAuth user-to-machine (U2M) authentication with Azure Databricks users.

In summary, use the following best practices when using OAuth secrets with service principals:

• Rotate secrets regularly
• Store secrets securely in Azure Key Vault or equivalent
• Never store secrets in plaintext
• Restrict access to secret storage
• Use the client credentials flow for OAuth M2M authentication

Proper management of OAuth secrets is critical to authenticate service principals to Azure Databricks securely.

 If you have any further questions, feel free to ask! 😊

[VJ3]

Thank you @Kaniz_Fatma for the response. I do have follow up questions.

- What kind of encryption is used to store OAUTH secret?

-  Is there any way OAUTH can be generated by someone else who is not a manager of that SPN? We need this as a part of segregation of duty

- Can we use OAUTH secret for non M2M authentication? 

- What is the purpose of .databrickscfg file? Can we avoid using it as someone can store Secret in plain text?

- Can we create multiple OAUTH Secret for single SPN?