cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Query on using secret scope for dbt-core integration with databricks workflow

Ram-Dev7
New Contributor

Hello all,

I am currently configuring dbt-core with Azure Databricks Workflow and using Azure Databricks M2M (Machine-to-Machine) authentication for this setup. I have the cluster ID and cluster secret ID stored in Databricks secret scope.

I am seeking advice on how to use these secrets in the `profiles.yml` file without exposing them as environment variables within Databricks. I need to ensure that these sensitive credentials remain secure while running through the Databricks workflow.

Has anyone successfully managed this setup or could provide any guidance on how to securely integrate these secrets into `profiles.yml` for dbt-core on Azure Databricks? Any inputs or best practices would be greatly appreciated.

Thank you!

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/oauth-m2m

2 REPLIES 2

Kaniz_Fatma
Community Manager
Community Manager

Hi @Ram-Dev7

For more detailed instructions, you can also refer to the official Microsoft Learn documentation.

Rishabh_Tiwari
Community Manager
Community Manager

Hi @Ram-Dev7 ,

Thank you for reaching out to our community! We're here to help you.

To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback not only helps us assist you better but also benefits other community members who may have similar questions in the future.If you found the answer helpful, consider giving it a kudo. If the response fully addresses your question, please mark it as the accepted solution. This will help us close the thread and ensure your question is resolved.

We appreciate your participation and are here to assist you further if you need it!

Thanks,
Rishabh