Query on using secret scope for dbt-core integration with databricks workflow

[Ram-Dev7]

Hello all,

I am currently configuring dbt-core with Azure Databricks Workflow and using Azure Databricks M2M (Machine-to-Machine) authentication for this setup. I have the cluster ID and cluster secret ID stored in Databricks secret scope.

I am seeking advice on how to use these secrets in the `profiles.yml` file without exposing them as environment variables within Databricks. I need to ensure that these sensitive credentials remain secure while running through the Databricks workflow.

Has anyone successfully managed this setup or could provide any guidance on how to securely integrate these secrets into `profiles.yml` for dbt-core on Azure Databricks? Any inputs or best practices would be greatly appreciated.

Thank you!

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/oauth-m2m

[Kaniz_Fatma]

Hi @Ram-Dev7, 

• Instead of storing secrets directly in profiles.yml, you can set environment variables for the client ID and client secret:

  export DATABRICKS_CLIENT_ID=<client-id>
  export DATABRICKS_CLIENT_SECRET=<client-secret>
  

• Refer to the appropriate profile and target YAML keys/values in profiles.yml for your Databricks SQL....

For more detailed instructions, you can also refer to the official Microsoft Learn documentation.

[mempl]

OP didnt wnat to expose environment variables:

"I am seeking advice on how to use these secrets in the `profiles.yml` file without exposing them as environment variables within Databricks. I need to ensure that these sensitive credentials remain secure while running through the Databricks workflow."

Is there another solution. like accessing secret scopes?

Best regards

[Rishabh_Tiwari]

Hi @Ram-Dev7 ,

Thank you for reaching out to our community! We're here to help you.

To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback not only helps us assist you better but also benefits other community members who may have similar questions in the future.If you found the answer helpful, consider giving it a kudo. If the response fully addresses your question, please mark it as the accepted solution. This will help us close the thread and ensure your question is resolved.

We appreciate your participation and are here to assist you further if you need it!

Thanks,
Rishabh