topic Re: What is the best practice for managing different environments (Staging vs Production) on Databricks? in Data Engineering

What is the best practice for managing different environments (Staging vs Production) on Databricks?

Anonymous — Mon, 21 Jun 2021 21:37:37 GMT

Should we create separate workspaces for Dev/Test/Prod ? Or should we have 1 workspace and create separate folders for Dev/Test/Prod?

Re: What is the best practice for managing different environments (Staging vs Production) on Databricks?

Srikanth_Gupta_ — Tue, 22 Jun 2021 14:51:55 GMT

as per my previous experience, its always good to have different workspaces for different envs, its easy to maintain and helps better with CICD pipeline as well, because lot of organizations provide deployment access to Developers in Dev env but not in UAT and Prod environments.

Re: What is the best practice for managing different environments (Staging vs Production) on Databricks?

Oliver_Angelil — Tue, 02 May 2023 11:02:02 GMT

@Srikanth Anumula we currently have 2 databricks workspaces: development and production. If the data on both are the same and not sensitive, is there any reason to have two workspaces? One could have one and simply have different git branches: feature-branches, devevelopment-branch, production-branch. I believe that might simplify things... .WDYT?

Re: What is the best practice for managing different environments (Staging vs Production) on Databricks?

harraz — Mon, 22 May 2023 22:55:48 GMT

Hi Srikanth,

Can you elaborate on your answer a bit more. We are new to databricks and we are not sure how can a developer work on a notebook that changes a table and at the same time make sure they don't make changes to production tables unless the code is tested and approved. Is there a best practice page that shows how to setup workspaces for this use case? And when I say workspace I mean the the personal workspace where developers have their own notebooks and hive metastore

thanks