How are you separating dev, staging and prod in Unity Catalog without duplicating everything?

[Islam_hoti]

Hi everyone, We are trying to settle on an environment strategy and keep going back and forth, so I would like to hear how other teams actually landed on this rather than what the reference architecture suggests. Our situation. One Databricks account, one metastore per region. Right now we have a single workspace where everything runs, which is obviously not sustainable. The two options on the table are separate catalogs in the same workspace, so dev_sales, staging_sales, prod_sales, or fully separate workspaces per environment with separate catalogs in each. Catalogs are simpler to operate and make it trivial to read prod data from dev, which our analysts like. But the isolation is only as strong as the grants, and one wrong GRANT gives someone write access to prod. Separate workspaces give real isolation, but then promoting a pipeline means keeping job definitions, cluster policies, secrets and permissions in sync across three places, and cross environment reads become awkward. Questions for anyone who has run either at scale. Which model did you pick, and would you pick it again? If you went with catalogs in one workspace, how do you stop dev work from accidentally writing to prod? Are grants alone enough in practice, or did you need something more? If you went with separate workspaces, how painful is the sync? Are Asset Bundles carrying enough of it, or are you still maintaining Terraform alongside? How do you handle test data? Copy a subset of prod down to dev, read prod directly from dev with read only grants, or generate synthetic data? And a specific one that keeps biting us: what do you do about a pipeline in dev that needs to read a prod table? Grant cross catalog read, or force a copy? Interested in the messy real answers, not just the clean ones. Thanks.

[Satyasai]

I would recommend the following

Workspaces: Deploy 3 Workspaces per region (Dev, Staging, Prod).

Catalogs: Create environment-specific catalogs inside Unity Catalog (dev_catalog, staging_catalog, prod_catalog).

Workspace-Catalog Bindings:

Bind dev_catalog to Dev Workspace (Full Access).

Bind prod_catalog to Prod Workspace (Full Access) and Dev Workspace (READ ONLY access for debugging/testing).

Deployment: Use Terraform for initial workspace and metastore setup, and
Databricks Asset Bundles (DABs) in Git CI/CD pipelines to promote code, jobs, and workflows across environments.

Additional Information -

Strict Service Principal Ownership: The production catalog (prod_catalog) must be owned by a dedicated CI/CD Service Principal (sp-prod-deployer). No human user—not even Senior Data Engineers—should have WRITE, MODIFY, or OWNERSHIP privileges in prod_catalog.

Read-Only Human Access: Data Engineers get SELECT access on prod_catalog for debugging, but zero CREATE or ALTER capabilities.

ABAC & Managed Identity Separation: Bind separate storage credentials / Azure Managed Identities / AWS IAM roles to the catalogs. The storage credential mounted to prod_catalog should explicitly deny write operations to dev user identities at the cloud IAM level.

[balajij8]

@Islam_hoti

You can keep separate workspaces per environment with one catalog namespace per env (prod.sales, staging.sales, dev.sales) managed through bundles DAB. Generate synthetic data for unit tests and use an anonymized subset of prod to the dev catalog on a schedule for integration tests. You can use Delta Sharing to expose specific prod tables read only to other workspaces using a dedicated service principal that gives a isolation boundary without copy. You can start with this structure.

[data_pulse]

From experience of working in diff projects, separate workspaces + environment catalogs + governed Prod reads from Dev is a practical combination.

• Environment isolation: Dev/Test/Prod workspaces with dev_*, test_*, prod_* catalogs. Personal development uses dev_<username>_ schemas within Dev.
• Infrastructure and access: Terraform provisions workspaces, catalog bindings, compute policies, and baseline access. Entra ID groups separate Prod/Non-Prod role. Unity Catalog grants control USE CATALOG, USE SCHEMA, SELECT, MODIFY etc. Restrict MANAGE and ownership to designated administrators.
• Service principals: Separate Prod and NonProd service principals for deployment and job execution, with scoped permissions and environment-specific secret scopes.
• CI/CD: GitHub Actions + DAB, shared workload definitions, and target-specific variables. PR validation, automated Dev/Test deployment, and controlled Prod deployment behind GitHub Environment approval gates.
• Prod reads from Dev: Cross Catalog Read-only access for model training, pipeline validation, and investigation. Dev identities receive the required read privileges, without Prod MODIFY, MANAGE, or ownership.
• Licensing and ABAC: license column + LPI account groups, with row filtering through IS_ACCOUNT_GROUP_MEMBER(). Column masks and ABAC policies protect sensitive fields, catalog visibility does not grant unrestricted data access.
• LPI automation: Capture discovers table/license mappings and  reconciles SELECT grants and row filters. Keep these dynamic silver/gold permissions outside Terraform/DAB grant management to avoid conflicting updates.
• Test data: Small, deterministic fixtures for unit tests, temporary dev schemas and mock source tables for integration tests, cleaned up post the runs. Acceptance tests validate deployed pipelines in Test. For model training, pipeline validation, and debugging, use read-only Prod access governed by LPI groups, row filters, and column masks.

The right approach depends on Use case + Constraints + Maintenance effort + Resources + Cost etc. Apply the best practices that fit, make the solution robust and maintainable. Avoid chasing a "gold standard" that adds complexity without real value.

[Khasim_1 · Accepted solution]

Hi @Islam_hoti,

This is a classic "architectural trade-off." Here is the messy reality of how we handle it:

1. Why we chose Separate Workspaces: We prioritize security boundaries. Even if grants are correct, separate workspaces prevent "blast radius" issues (like accidental clicks or service principal misconfigurations) from touching Production.
2. How we fix the "Sync Pain": We use Databricks Asset Bundles (DABs) for everything. If the job definition isn't in our bundle code, it doesn't exist. We treat environment synchronization as a CI/CD problem, not a manual task.
3. The Dev-to-Prod Bridge: We never copy data. Instead, we use Catalog-level Read-Only grants. We allow the Dev workspace to query the Production catalog with SELECT permissions only. It gives analysts the visibility they need without the risk of accidental writes.
4. Test Data Strategy: We use synthetic data or small subsets in Dev to validate logic, and rely on DLT expectations to "fail fast." We don't spend time moving production data to dev; we move the code to the data.

The Honest Verdict: If you are a small team, one workspace is fine. But for enterprise-grade maturity, go with separate workspaces. The operational cost of syncing is a one-time setup; the risk of a "write to production" incident is a career-defining problem you want to avoid.

[coolbeans201]

Many different ways to go about this, but I think the key is a separate catalog per environment. If you have different workspaces for Dev/QA and Prod, then making sure the lower environments only have read access to Prod (if wanting to test against actual data in order to avoid duplication) is a good strategy as well.

[Coffee77]

Here is, in summary, how I am facing that issue in some real projects in Deloitte:

Unity Catalog Metastore
│
├── Workspace DEV
│ └── sales_dev
│ └── finance_dev
│ └── customer_dev
│
├── Workspace QA
│ └── sales_qa
│ └── finance_qa
│ └── customer_qa
│
├── Workspace STAGE
│ └── sales_stage
│ └── finance_stage
│ └── customer_stage
│
└── Workspace PROD
└── sales_prod
└── finance_prod
└── customer_prod

For me, it is key to separate environments by workspace for better isolation. Having said this, all four workspaces are attached to the same Unity Catalog metastore. This gives you centralized governance, identities, permissions, lineage, auditing, storage credentials, and other UC capabilities. Databricks explicitly supports attaching multiple workspaces in the same region to a single metastore.

Another key point is that catalogs provide the data environment boundary. A logical data domain such as sales is deployed as four independent catalogs, with specific suffixes as you can see above. So, take special care in binding or assigning proper catalogs to workspaces as evidently catalog naming alone does not provide workspace isolation.

Inside each catalog, you can keep exactly the same schema and table structure and create schemas and tables similar to: sales_dev.bronze.orders, sales_dev.silver.order, sales_dev.gold.orders. In nay case, I strongly recommend deploy data objects by using code, never create them manually, you can use custom code, dbt, etc.

I hope this helps.