topic Re: Secrete management in Administration & Architecture

Secrete management

Snoonan — Thu, 09 May 2024 09:00:13 GMT

Hi all,

I am trying to use secrets to connect to my Azure storage account. I want to be able to read the data form the storage account using a pyspark notebook.

Has anyone experience setting up such a connection or has good documentation to do so?

I have come across Databricks managed secrets and Azure key vault managed secrets. I am not sure of the difference. Could anyone clarify the difference?

Thanks,

Sean

Re: Secrete management

AmanSehgal — Thu, 09 May 2024 12:49:32 GMT

Have you tried using Azure key vault backed secret scope?

Secret scopes - Azure Databricks | Microsoft Learn

Re: Secrete management

DonatienTessier — Mon, 13 May 2024 08:36:48 GMT

Hi Sean,

There are two ways to handle secret scopes:

databricks-backed scopes: scope is related to a workspace. You will have to handle the update of the secrets.
Azure Key Vault-backed scopes: scope is related to a Key Vault. It means than you configure the access to KV using a scope and then you will be able to access the secrets stored in the KV (if you configured properly the access first).

The security best practices is to use an Azure Key Vault-backed scopes. If there is some rotation policies activate, it will be handle.

Nevertheless, if you need to access to a storage account (in case of ADLS), it is better if you can use an access connector rather than using the access key, for example.

I hope it is clearer now 🙂

Re: Secrete management

Snoonan — Wed, 22 May 2024 11:03:22 GMT

Hi @DonatienTessier ,

This is very clear. Thank you!