How to manage two separate projects ?

[chexa_Wee]

Hi all, I am managing one project in Databricks, with one more coming soon. Can anyone guide me on how to use Unity Catalog or any other method for this?"

[MariuszK]

Hi,

What do you need to know?

[chexa_Wee]

Hi,I want to create separate databases for two projects, where I can organize bronze, silver, and gold layers while keeping all data separate. Could you please guide me on the best approach for this?

[MadhuB · Accepted solution]

@chexa_WeeIf both projects (databricks) are in different cloud regions, you can have 2 metastores which is a straight forward implementation. Otherwise, below are the possibilities.

Easiest Approach - You can’t have two metastores unless they are in separate cloud regions. In your case, I recommend creating a catalog for each of Bronze, Silver, and Gold, and then setting up project-specific schemas and objects within those catalogs.

Complex with more admin maintenace
If you have more than one storage, then ofcourse, you can have a setup like below. However, you end up with more catalogs and lot maintenance on admin side. 

Storage1
Catalog11(Bronze) -- Schema -- Objects
Catalog12(Silver) -- Schema -- Objects
Catalog13(Gold) -- Schema -- Objects

Storage2
Catalog21(Bronze) -- Schema -- Objects
Catalog22(Silver) -- Schema -- Objects
Catalog23(Gold) -- Schema -- Objects

Having more catalogs and storage layers can also lead to more complex data governance and maintenance efforts. Please mark this as a solution if it addresses your question.

[chexa_Wee]

Given the complexity and increased administrative maintenance, I need to adopt the approach of creating separate storage accounts (Storage 1 and Storage 2). Could you explain how to implement this effectively? With more projects on the way, I need to set up separate databases with bronze, silver, and gold layers to ensure data cleanliness, maintain data integrity, and prevent data from getting mixed together.

[MariuszK]

I don't know border context of your data and project, but you can create a separatr catalog for each project or you can create catalogs: bronze, silver, and gold and separate schema for each of projects.

The right design depends on a need to access data and purpose of different catalogs.

[chexa_Wee]

Can you explain me how to create separate catalog for each project? Thank you

[mnorland]

There are a wide variety of needs that need to be considered such as governance, compute and storage.  This depends on the size of your projects.