Issue with Generic DLT Pipeline Handling Multiple BUs
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago
We are implementing a data ingestion framework where data flows from a foreign catalog (source) to a raw layer (Delta tables) and then to a bronze layer (DLT streaming tables). Currently, each Business Unit (BU) has a separate workflow and DLT pipeline (e.g., BU1 → Workflow1 & DLT1, BU2 → Workflow2 & DLT2).
To improve efficiency, we aim to create a generic workflow and DLT pipeline that supports multiple BUs without maintaining separate workflows. While workflow-level adjustments are feasible, the challenge arises at the DLT pipeline level:
- If we modify the path where table metadata is stored, tables get dropped from the catalog, impacting the pipeline.
- We need a way to segregate BU-specific tables within the same DLT pipeline while ensuring they persist across updates.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago
You can create separate schemas within the same catalog for each BU. For example, you can have schemas like BU1_schema
, BU2_schema
, etc., within the same catalog.
By using Unity Catalog, you can segregate BU-specific tables within the same DLT pipeline
https://docs.databricks.com/en/delta-live-tables/unity-catalog.html
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
3 weeks ago
Thanks, @NandiniN, for your response. The requirement here is not about separating the schema, as we are already handling that with direct publishing mode.
The question is whether we can use a single DLT pipeline to execute multiple groups of tables without causing previously created tables to be dropped from Unity Catalog.
For example, in the first DLT run, tables 1-50 are created in Unity Catalog. However, when we use the same DLT pipeline for another set of tables (51-100), the previously created tables (1-50) are dropped, and only tables 51-100 remain in Unity Catalog. We want to prevent this dropping behavior.

