How to enable / setup OAuth for DBT with Databricks

Th0rs7en
New Contributor II

‎05-01-2025 02:43 AM

I tried configure / setup DBT to authenticate with OAuth to Databricks, following the tutorials https://community.databricks.com/t5/technical-blog/using-dbt-core-with-oauth-on-azure-databricks/ba-... and https://docs.databricks.com/aws/en/partners/prep/dbt 

But unfortunately `use oauth` was not offered as option in the dbt init dialog - unlike in the tutorial.
Also adding the authentication "

auth_type: oauth
client_id: "9729e77e-ba94-4d53-8cfb-bb609f43f881"

Did not work.

How to debug / analyze / setup to authenticate with OAuth

0 Kudos

Louis_Frolio
Databricks Employee
Databricks Employee

‎05-01-2025 08:36 AM

Here are some things to consider:

Debugging and Setting Up OAuth Authentication with dbt and Databricks

Steps to Analyze and Setup Authentication: 1. Verify Credentials and Configuration: - Double-check the auth_type: oauth is correctly specified, as well as the client ID. Ensure these are properly formatted. - Ensure sensitive information like secrets and tokens are set as environment variables instead of being hardcoded in the profiles.yml.
  1. Using OAuth Tokens:
    • Databricks recommends using OAuth tokens as the secure method for authentication.
    • Confirm that your Databricks instance has OAuth enabled as specified in the documentation.
  2. dbt init Dialog:
    • If the use oauth option is not displayed in the dbt init dialog, it could indicate that either the version of dbt-databricks is outdated or that the OAuth setup on Databricks is incomplete.
    • Ensure you are using dbt-databricks version 1.8.0 or greater as recommended.
  3. Testing Connection:
    • Use the dbt debug command to confirm that the connection details are correct and the profiles file is valid.
    • For debugging errors related to authentication, the console output during dbt debug will provide insights.
  4. Environment and Prerequisites:
    • Confirm Python 3.7+ is installed along with utilities like pipenv for virtual environments.
    • Ensure correct host and http_path settings in the profiles.yml file, derived from your Databricks workspace (Compute or SQL Warehouse settings).
  5. External Tutorials:
    • The community tutorial you referenced might contain additional steps specific to Azure Databricks. However, since we cannot access its content, ensure your setup aligns with the generic Databricks OAuth process outlined in the Databricks documentation.
Additional Tips: - If you are still unable to authenticate, check the compatibility between your dbt-databricks package and your Databricks workspace. - For further guidance, consult the Connect to dbt Core Databricks documentation. 
 
Hope this helps, Big Roux.
 
0 Kudos