cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Setting Up Databricks Workspace

Loki
New Contributor III

 

Hi Team, 

Need some advice on setting up a single workspace or multiple workspaces.
Please read below for the problem statement - 
There is an ADLS account says ADLS1 which has a root folder say root1
This root1 folder has multiple sub folders
team1
team2
team3

We want to implement a strategy so that team1 cannot access data from team2, team3 and the same is applicable for teams 2 and 3. 

Should we create 3 workspaces here or use single workspace with 3 folders?
If we decide to use later, would the teams be able to implement their service control separately in three different repositories using Repos ?

PS - In one of the previous projects we were able to use a single workspace with 3 folders for each of the dev, qa and production environments.
However, we were not using repos there.

 

 

1 ACCEPTED SOLUTION

Accepted Solutions

karthik_p
Esteemed Contributor

@Loki As per bets practices, best way is go with single workspace1 for multiple teams (Dev), workspace 2- QA,  workspace3 - Prd or based on business unit .

View solution in original post

6 REPLIES 6

karthik_p
Esteemed Contributor

@Loki you need to consider 2 things 1. environment wise segregation (dev,qa, prd) --> this will be good strategy for CICD and testing wise as in dev you can play with your data and customize according to needs before going to prod

2. coming data security (teams1, team2, teams 3) --> go with unity catalog ( best solution for governance)

Loki
New Contributor III

Thanks Karthik, as far as I understand unity catalog would be useful from delta lake perspective which is the destination of our ETL pipeline. 
my question was about data protection on the source layer. We have an ADLS account and container which could have separate folders for different teams. Should be create a single workspace and have multiple folders inside it and use ACL to implement the security ?

karthik_p
Esteemed Contributor

@Loki Are you talking about storage that is created during workspace which is root storage, i believe that is by default picked by azure during workspace implementation, where root storage is create (which includes DBFS root). for that teams segregation is not needed. if you have defined workspaces in dev,qa, prod. each workspace will have different root storage. this is where your hive legacy metadata gets stored.

if you are preferring to store u r data in external ( data, metadata), that is where ADLSG2 comes into picture. can you be specific on what is source layer please 

Loki
New Contributor III

As mentioned above I am talking about ADLS Gen 2. It's a typical ETL scenario. 

Let me reframe my question, can we get away by creating seperate workspace for separate team or can we use single workspace?

karthik_p
Esteemed Contributor

@Loki As per bets practices, best way is go with single workspace1 for multiple teams (Dev), workspace 2- QA,  workspace3 - Prd or based on business unit .

Anonymous
Not applicable

Hi @Loki 

Thank you for posting your question in our community! We are happy to assist you.

 

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

 

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance! 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group