Authenticate Databricks REST API and access delta tables from external web service.

[sensanjoy]

Hi All,

We do have a requirement to access delta tables from external web service(Web UI). Presently we have tested it through jdbc connection and authenticated using PAT:

Ex. jdbc:spark://[DATABRICKS_HOST]:443/default;transportMode=http;ssl=1;httpPath=[DATABRICKS_HTTP_PATH];AuthMech=3;UID=token;PWD=<personal-access-token>

Now, we are planning to implement it through Service Principal with Azure AD token that can expire within 1 hour.

My question is:

1. Do we need any management token other than Azure AD token(created for Service Principal) to access tables from external web service?

2. If yes, could you please share some example link.

3. Can we use this Service Principal and Azure AD token to create new DB pipeline (jenkins CI/CD) : Existing is(Azure Resource Token+ PAT token)

4. Any 'Best Practice' to access delta tables from external web service.

[Debayan]

Hi, For JDBC authentication, you can refer to https://learn.microsoft.com/en-us/azure/databricks/integrations/jdbc-odbc-bi#--authentication-requir...

Please let us know if this helps. 

Also, please tag @Debayan​ with your next response so that I will be notified. Thanks!

[sensanjoy]

Thanks for the reply @Debayan Mukherjee​ 

Looking at that link shared by you which is somehow related to my first question, now I am bit confused after checking the connection string for JDBC :

Does it mean we need both Azure AD token and PAT to authenticate with the help of Service Principal? if yes, then I guess PAT is created for specific user not for any Service Principal!!

[sensanjoy]

Hi @Suteja Kanuri​ , could you please help me with above queries.

[Anonymous]

@Sanjoy Sen​ :

1. When using a Service Principal with Azure AD token to access Delta tables from an external web service, you do not need any additional management token. The Azure AD token should be sufficient for authentication and authorization purposes.
2. Here's an example link that demonstrates how to authenticate and authorize access to Delta tables using a Service Principal and Azure AD token:
3. Yes, you can use a Service Principal and Azure AD token to create a new DB pipeline (Jenkins CI/CD) instead of using the existing Azure Resource Token and PAT token. You would need to configure the pipeline to use the appropriate authentication mechanism and provide the necessary credentials for the Service Principal and Azure AD token.
4. Some best practices for accessing Delta tables from external web services include:

• Always use secure connections (e.g., HTTPS) to protect sensitive data and credentials.
• Use a Service Principal with Azure AD token instead of a PAT token for improved security.
• Limit access to Delta tables to only the necessary users and roles.
• Monitor access to Delta tables and audit activity regularly to detect and respond to potential security incidents.

[sensanjoy]

@Suteja Kanuri​ Thanks a lot.

Can I take the first point as: If the AAD token is part of the workspace with adequate privilege then it works otherwise to authorize, we need some PAT token on behalf of it and using PAT we may control the lifetime of the token otherwise it would be 1 hour(based on Azure AD token's )!!

Could you please share the link as mentioned over point 2.

[Anonymous]

@Sanjoy Sen​ :

Link: https://docs.databricks.com/dev-tools/api/latest/authentication.html and https://docs.databricks.com/dev-tools/service-principals.html

[Anonymous]

@Sanjoy Sen​ :

Yes, that's correct. If the Azure AD token being used has the necessary permissions to access the Delta tables, then you don't need any additional management token. However, if the Azure AD token does not have the necessary permissions, you would need to authenticate using a personal access token (PAT) that has the required permissions.

Additionally, when using the Azure AD token, the token lifetime is determined by the Azure AD settings, which is usually 1 hour. But if you use a PAT, you have control over the lifetime of the token.