Unity catalog implementation

prad18
Databricks Partner

Hello Databricks Community,

We are in the process of planning a Unity Catalog implementation for our organization, and I'd like to seek input on some architectural decisions. We're considering various approaches to workspace separation, storage account allocation, and Azure subscription management. I'd greatly appreciate insights from those who have experience with similar implementations.

I have already done some research through most of the Data + AI Summit videos, Databricks blogs & best practices but haven't seen anything detailed implementation regarding unity catalog.

Here are the key questions we're grappling with:

1. Workspace Separation Strategy : 

Development and Staging environments -> Single Workspace
Production environment -> Separate Workspace
What are the advantages and disadvantages of this approach?

2. Storage Account Allocation :

Should we use separate Azure storage accounts for:

  1. Unity Catalog metastore
  2. Actual data (landing zone)

What are the implications of this approach, and what factors should we consider in making this decision?

3. Azure Subscription and Metastore Architecture :

Should we maintain separate Azure subscriptions for Development and Production environments? ( i think we should)
If we opt for separate subscriptions:
a) Where should we create the Unity Catalog metastore(s)?
b) Should we use a single metastore in the Production subscription, or create separate metastores for each environment (one in Production, one in Development)?

What are the pros and cons of each approach, considering factors such as data governance, security, and operational efficiency?"

If you've tackled similar decisions in your Unity Catalog implementation, what approach did you take? What worked well, and what challenges did you face?

Thank you in advance for your insights and advice!"