Hey mbanxp!
The most scalable and maintainable orchestration pattern for multi-tenant medallion architectures in Databricks is to build independent pipelines per table for all clients, with each pipeline parameterized by client/tenant.
Why this approach?
- Centralizes business logic for each table (reduces code duplication).
- Makes onboarding new clients easy—just add configuration, don't duplicate pipeline code.
- Scales well as data and client count grow.
- Fits perfectly with Databricks Workflows and Delta Live Tables (DLT), which support parameterized, multi-tenant pipelines and robust orchestration.
- Unity Catalog provides strong data isolation and governance at the client level, even when sharing pipelines.
Platform Features Enabling This Pattern:
- Databricks Workflows: Orchestrate parameterized, multi-tenant pipelines.
- Delta Live Tables (DLT): Declaratively define ETL flows partitioned by client.
- Unity Catalog: Fine-grained access control and catalog/schema separation per client.
Extra tips:
Leverage partitioning and schema separation by client within each layer, and use centralized pipelines to tune job frequencies and resource usage.
Summary:
Organizing by per-table, multi-tenant pipelines is Databricks’ best practice for efficient, standardized, and easily-governed medallion data flows at scale.
I hope this helps.
Best,
Sarah
