topic Re: How to manage two separate projects ? in Data Engineering

How to manage two separate projects ?

chexa_Wee — Tue, 25 Mar 2025 07:19:17 GMT

Hi all, I am managing one project in Databricks, with one more coming soon. Can anyone guide me on how to use Unity Catalog or any other method for this?"

Re: How to manage two separate projects ?

MariuszK — Tue, 25 Mar 2025 09:55:58 GMT

Hi,

What do you need to know?

Re: How to manage two separate projects ?

chexa_Wee — Tue, 25 Mar 2025 11:50:44 GMT

Hi,I want to create separate databases for two projects, where I can organize bronze, silver, and gold layers while keeping all data separate. Could you please guide me on the best approach for this?

Re: How to manage two separate projects ?

MariuszK — Tue, 25 Mar 2025 13:41:57 GMT

I don't know border context of your data and project, but you can create a separatr catalog for each project or you can create catalogs: bronze, silver, and gold and separate schema for each of projects.

The right design depends on a need to access data and purpose of different catalogs.

Re: How to manage two separate projects ?

mnorland — Tue, 25 Mar 2025 14:47:40 GMT

There are a wide variety of needs that need to be considered such as governance, compute and storage. This depends on the size of your projects.

Re: How to manage two separate projects ?

MadhuB — Tue, 25 Mar 2025 16:17:10 GMT

@chexa_WeeIf both projects (databricks) are in different cloud regions, you can have 2 metastores which is a straight forward implementation. Otherwise, below are the possibilities.

Easiest Approach - You can’t have two metastores unless they are in separate cloud regions. In your case, I recommend creating a catalog for each of Bronze, Silver, and Gold, and then setting up project-specific schemas and objects within those catalogs.

Complex with more admin maintenace
If you have more than one storage, then ofcourse, you can have a setup like below. However, you end up with more catalogs and lot maintenance on admin side.

Storage1
Catalog11(Bronze) -- Schema -- Objects
Catalog12(Silver) -- Schema -- Objects
Catalog13(Gold) -- Schema -- Objects

Storage2
Catalog21(Bronze) -- Schema -- Objects
Catalog22(Silver) -- Schema -- Objects
Catalog23(Gold) -- Schema -- Objects

Having more catalogs and storage layers can also lead to more complex data governance and maintenance efforts. Please mark this as a solution if it addresses your question.

Re: How to manage two separate projects ?

chexa_Wee — Wed, 26 Mar 2025 03:35:40 GMT

Given the complexity and increased administrative maintenance, I need to adopt the approach of creating separate storage accounts (Storage 1 and Storage 2). Could you explain how to implement this effectively? With more projects on the way, I need to set up separate databases with bronze, silver, and gold layers to ensure data cleanliness, maintain data integrity, and prevent data from getting mixed together.

Re: How to manage two separate projects ?

chexa_Wee — Wed, 26 Mar 2025 04:06:16 GMT

Can you explain me how to create separate catalog for each project? Thank you