- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-14-2022 02:51 PM
We have a development and a production data lake. Is it possible to have a production or development cluster access only respective mounts using init scripts?
- Labels:
-
Azure
-
Azure data lake
-
AzureStorage
-
Mount
-
Mount Point
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-14-2022 03:25 PM
Yes it is possible. Additionally mount is permanent and done in dbfs so it is enough to run it one time.
you can have for example following configuration:
- In Azure you can have 2 databricks workspace,
- cluster in every workspace can have env variable is it DEV or PROD,
- Key vault can be common for both,
- Common repo but production/master branch in PROD,
- Common infrastructure folder which mounts folders, take settings from key vaults and depends is it DEV or PROD choose needed bucket/container
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-14-2022 03:25 PM
Yes it is possible. Additionally mount is permanent and done in dbfs so it is enough to run it one time.
you can have for example following configuration:
- In Azure you can have 2 databricks workspace,
- cluster in every workspace can have env variable is it DEV or PROD,
- Key vault can be common for both,
- Common repo but production/master branch in PROD,
- Common infrastructure folder which mounts folders, take settings from key vaults and depends is it DEV or PROD choose needed bucket/container
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-14-2022 03:34 PM
Hi Hubert. Thanks!
Can you please show me an example for how I can use environment variables to choose a bucket/container?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-15-2022 12:28 AM
https://docs.databricks.com/clusters/configure.html#environment-variables
Alternatively you can also use a config file (.json, .conf, ...) which you upload to your workspace.
Depending on the environment, another container is selected.