Secrete management

[Snoonan]

Hi all,

I am trying to use secrets to connect to my Azure storage account. I want to be able to read the data form the storage account using a pyspark notebook.

Has anyone experience setting up such a connection or has good documentation to do so?

I have come across Databricks managed secrets and Azure key vault managed secrets. I am not sure of the difference. Could anyone clarify the difference?

Thanks,

Sean

[AmanSehgal]

Have you tried using Azure key vault backed secret scope?

Secret scopes - Azure Databricks | Microsoft Learn

[DonatienTessier · Accepted solution]

Hi Sean,

There are two ways to handle secret scopes:

• databricks-backed scopes: scope is related to a workspace. You will have to handle the update of the secrets.
• Azure Key Vault-backed scopes: scope is related to a Key Vault. It means than you configure the access to KV using a scope and then you will be able to access the secrets stored in the KV (if you configured properly the access first).

The security best practices is to use an Azure Key Vault-backed scopes. If there is some rotation policies activate, it will be handle.

Nevertheless, if you need to access to a storage account (in case of ADLS), it is better if you can use an access connector rather than using the access key, for example.

I hope it is clearer now 🙂

[Snoonan]

Hi @DonatienTessier ,

This is very clear. Thank you!