Azure Databricks SQL Execution API Authentication

[Sid1805]

Hi Team,

If a cloud application wants to read some data from Databricks we realize that Azure Databricks can expose its Tables via REST API.

For its authentication what is the most recommended method - I see we can have PAT token tagged to a Service Principal seemingly most secure ?

Also in the documentation I am confused with the below statement - ( as Oauth is for authorization and not authentication ?? )

***************

As a security best practice when you authenticate with automated tools, systems, scripts, and apps, Databricks recommends that you use OAuth tokens.

If you use personal access token authentication, Databricks recommends using personal access tokens belonging to service principals instead of workspace users. To create tokens for service principals, see Manage tokens for a service principal.

*************

Document Link - https://docs.databricks.com/en/sql/admin/sql-execution-tutorial.html

[Kaniz]

Hi @Sid1805, 

When it comes to authenticating with Azure Databricks, there are several methods available. Let’s break down the options:

1. OAuth Tokens:
   • Recommended for Automation: OAuth tokens are a secure choice for authenticating automated tools, systems, scripts, and applications.
   • Authorization vs. Authentication: You’re right that OAuth is primarily used for authorization, but in this context, it’s also used for authentication. OAuth tokens verify the identity of the caller.
   • Scoping and Expiration: OAuth tokens offer better server-side session invalidation and scoping. They have faster expiration times compared to personal access tokens (PATs).
   • Use Cases: OAuth tokens are suitable for both Databricks account-level functionality and workspace-level functionality.
   • How to Create: You can create OAuth tokens for Databricks account admin users or service principals.
   • Link: OAuth Authentication for Service Principals

2. Personal Access Tokens (PATs):
   • Workspace-Level Credentials: PATs are commonly used credentials for resources and operations within a specific Databricks workspace.
   • Service Principals: For enhanced security, Databricks recommends using PATs associated with service principals rather than workspace users.
   • Limitations: Each PAT works only for a single Databricks workspace, and users are limited to 600 PATs per workspace.
   • Account-Level Functionality: However, if you need to automate Databricks account-level functionality, you cannot use PATs. Instead, opt for OAuth tokens or account-level admin credentials (username and password).
   • How to Create: To generate a PAT, follow these steps:
     1. Click your Databricks username in the top bar.
     2. Select “User Settings” from the dropdown.
     3. Click “Developer.”
     4. Under “Access tokens,” click “Manage.”
     5. Generate a new token, optionally adding a comment and adjusting the lifetime.
     6. Save the token securely.
   • Link: Databricks Personal Access Tokens

In summary, OAuth tokens are versatile and recommended for automation, while PATs are workspace-specific and best suited for workspace-level tasks. Choose the method that aligns with your specific use case and security requirements. 🛡️

[Sid1805]

@Kaniz - Thanks for your response.

For Azure Databricks my basic question is if the SQL REST API only supports PAT Tokens currently or it also supports Oauth as well ?

[Ramakrishnan83]

Hi @Sid1805 ,

I am in the same situation as you. I am looking for guidance how to setup the authentication for REST API to connect with Databricks SQL. I started with PAT and used  the user id as "token" and pwd as PAT Token to connect with Databricks SQL. 

Now I need to change the authentication to Service Principals. I have setup the service principal and gave access to the databricks SQL. Now how do i configure the Application client ID and Secret in API/ Azure Kubernetes  Layer