Is it possible to mount different Azure Storage Accounts for different clusters in the same workspace?

Bhanu1
New Contributor III

‎03-14-2022 02:51 PM

We have a development and a production data lake. Is it possible to have a production or development cluster access only respective mounts using init scripts?

Hubert-Dudek
Databricks MVP

‎03-14-2022 03:25 PM

Yes it is possible. Additionally mount is permanent and done in dbfs so it is enough to run it one time.

you can have for example following configuration:

  • In Azure you can have 2 databricks workspace,
  • cluster in every workspace can have env variable is it DEV or PROD,
  • Key vault can be common for both,
  • Common repo but production/master branch in PROD,
  • Common infrastructure folder which mounts folders, take settings from key vaults and depends is it DEV or PROD choose needed bucket/container


My blog: https://databrickster.medium.com/

View solution in original post

Bhanu1
New Contributor III
In response to Hubert-Dudek

‎03-14-2022 03:34 PM

Hi Hubert. Thanks!

Can you please show me an example for how I can use environment variables to choose a bucket/container?

0 Kudos

-werners-
Esteemed Contributor III
In response to Bhanu1

‎03-15-2022 12:28 AM

https://docs.databricks.com/clusters/configure.html#environment-variables

Alternatively you can also use a config file (.json, .conf, ...) which you upload to your workspace.

Depending on the environment, another container is selected.