Query on using secret scope for dbt-core integration with databricks workflow

[Ram-Dev7]

Hello all,

I am currently configuring dbt-core with Azure Databricks Workflow and using Azure Databricks M2M (Machine-to-Machine) authentication for this setup. I have the cluster ID and cluster secret ID stored in Databricks secret scope.

I am seeking advice on how to use these secrets in the `profiles.yml` file without exposing them as environment variables within Databricks. I need to ensure that these sensitive credentials remain secure while running through the Databricks workflow.

Has anyone successfully managed this setup or could provide any guidance on how to securely integrate these secrets into `profiles.yml` for dbt-core on Azure Databricks? Any inputs or best practices would be greatly appreciated.

Thank you!

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/oauth-m2m

[mempl]

OP didnt wnat to expose environment variables:

"I am seeking advice on how to use these secrets in the `profiles.yml` file without exposing them as environment variables within Databricks. I need to ensure that these sensitive credentials remain secure while running through the Databricks workflow."

Is there another solution. like accessing secret scopes?

Best regards

[Rishabh_Tiwari]

Hi @Ram-Dev7 ,

Thank you for reaching out to our community! We're here to help you.

To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback not only helps us assist you better but also benefits other community members who may have similar questions in the future.If you found the answer helpful, consider giving it a kudo. If the response fully addresses your question, please mark it as the accepted solution. This will help us close the thread and ensure your question is resolved.

We appreciate your participation and are here to assist you further if you need it!

Thanks,
Rishabh