Query on using secret scope for dbt-core integration with databricks workflow

[Ram-Dev7]

Hello all,

I am currently configuring dbt-core with Azure Databricks Workflow and using Azure Databricks M2M (Machine-to-Machine) authentication for this setup. I have the cluster ID and cluster secret ID stored in Databricks secret scope.

I am seeking advice on how to use these secrets in the `profiles.yml` file without exposing them as environment variables within Databricks. I need to ensure that these sensitive credentials remain secure while running through the Databricks workflow.

Has anyone successfully managed this setup or could provide any guidance on how to securely integrate these secrets into `profiles.yml` for dbt-core on Azure Databricks? Any inputs or best practices would be greatly appreciated.

Thank you!

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/oauth-m2m

[Kaniz_Fatma]

Hi @Ram-Dev7, 

• Instead of storing secrets directly in profiles.yml, you can set environment variables for the client ID and client secret:

  export DATABRICKS_CLIENT_ID=<client-id>
  export DATABRICKS_CLIENT_SECRET=<client-secret>
  

• Refer to the appropriate profile and target YAML keys/values in profiles.yml for your Databricks SQL....

For more detailed instructions, you can also refer to the official Microsoft Learn documentation.