topic Re: Is it possible to mount different Azure Storage Accounts for different clusters in the same workspace? in Data Engineering

Is it possible to mount different Azure Storage Accounts for different clusters in the same workspace?

Bhanu1 — Mon, 14 Mar 2022 21:51:46 GMT

We have a development and a production data lake. Is it possible to have a production or development cluster access only respective mounts using init scripts?

Re: Is it possible to mount different Azure Storage Accounts for different clusters in the same workspace?

Hubert-Dudek — Mon, 14 Mar 2022 22:25:54 GMT

Yes it is possible. Additionally mount is permanent and done in dbfs so it is enough to run it one time.

you can have for example following configuration:

In Azure you can have 2 databricks workspace,
cluster in every workspace can have env variable is it DEV or PROD,
Key vault can be common for both,
Common repo but production/master branch in PROD,
Common infrastructure folder which mounts folders, take settings from key vaults and depends is it DEV or PROD choose needed bucket/container

Re: Is it possible to mount different Azure Storage Accounts for different clusters in the same workspace?

Bhanu1 — Mon, 14 Mar 2022 22:34:37 GMT

Hi Hubert. Thanks!

Can you please show me an example for how I can use environment variables to choose a bucket/container?

Re: Is it possible to mount different Azure Storage Accounts for different clusters in the same workspace?

-werners- — Tue, 15 Mar 2022 07:28:12 GMT

https://docs.databricks.com/clusters/configure.html#environment-variables

Alternatively you can also use a config file (.json, .conf, ...) which you upload to your workspace.

Depending on the environment, another container is selected.