topic Re: Setting Up Databricks Workspace in Get Started Discussions

Setting Up Databricks Workspace

Loki — Fri, 30 Jun 2023 09:56:40 GMT

Hi Team,

Need some advice on setting up a single workspace or multiple workspaces.
Please read below for the problem statement -
There is an ADLS account says ADLS1 which has a root folder say root1
This root1 folder has multiple sub folders
team1
team2
team3

We want to implement a strategy so that team1 cannot access data from team2, team3 and the same is applicable for teams 2 and 3.

Should we create 3 workspaces here or use single workspace with 3 folders?
If we decide to use later, would the teams be able to implement their service control separately in three different repositories using Repos ?

PS - In one of the previous projects we were able to use a single workspace with 3 folders for each of the dev, qa and production environments.
However, we were not using repos there.

Re: Setting Up Databricks Workspace

karthik_p — Sat, 01 Jul 2023 20:29:23 GMT

@Loki you need to consider 2 things 1. environment wise segregation (dev,qa, prd) --> this will be good strategy for CICD and testing wise as in dev you can play with your data and customize according to needs before going to prod

2. coming data security (teams1, team2, teams 3) --> go with unity catalog ( best solution for governance)

Re: Setting Up Databricks Workspace

Anonymous — Sun, 02 Jul 2023 02:52:59 GMT

Hi @Loki

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!

Re: Setting Up Databricks Workspace

Loki — Sun, 02 Jul 2023 09:04:58 GMT

Thanks Karthik, as far as I understand unity catalog would be useful from delta lake perspective which is the destination of our ETL pipeline.
my question was about data protection on the source layer. We have an ADLS account and container which could have separate folders for different teams. Should be create a single workspace and have multiple folders inside it and use ACL to implement the security ?

Re: Setting Up Databricks Workspace

karthik_p — Sun, 02 Jul 2023 15:34:01 GMT

@Loki Are you talking about storage that is created during workspace which is root storage, i believe that is by default picked by azure during workspace implementation, where root storage is create (which includes DBFS root). for that teams segregation is not needed. if you have defined workspaces in dev,qa, prod. each workspace will have different root storage. this is where your hive legacy metadata gets stored.

if you are preferring to store u r data in external ( data, metadata), that is where ADLSG2 comes into picture. can you be specific on what is source layer please

Re: Setting Up Databricks Workspace

Loki — Sun, 02 Jul 2023 16:27:17 GMT

As mentioned above I am talking about ADLS Gen 2. It's a typical ETL scenario.

Let me reframe my question, can we get away by creating seperate workspace for separate team or can we use single workspace?

Re: Setting Up Databricks Workspace

karthik_p — Sun, 02 Jul 2023 17:56:13 GMT

@Loki As per bets practices, best way is go with single workspace1 for multiple teams (Dev), workspace 2- QA, workspace3 - Prd or based on business unit .