cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Issue with Generic DLT Pipeline Handling Multiple BUs

RamanBajetha
New Contributor II

We are implementing a data ingestion framework where data flows from a foreign catalog (source) to a raw layer (Delta tables) and then to a bronze layer (DLT streaming tables). Currently, each Business Unit (BU) has a separate workflow and DLT pipeline (e.g., BU1 → Workflow1 & DLT1, BU2 → Workflow2 & DLT2).

To improve efficiency, we aim to create a generic workflow and DLT pipeline that supports multiple BUs without maintaining separate workflows. While workflow-level adjustments are feasible, the challenge arises at the DLT pipeline level:

  • If we modify the path where table metadata is stored, tables get dropped from the catalog, impacting the pipeline.
  • We need a way to segregate BU-specific tables within the same DLT pipeline while ensuring they persist across updates.
2 REPLIES 2

NandiniN
Databricks Employee
Databricks Employee

You can create separate schemas within the same catalog for each BU. For example, you can have schemas like BU1_schema, BU2_schema, etc., within the same catalog.

By using Unity Catalog, you can segregate BU-specific tables within the same DLT pipeline

https://docs.databricks.com/en/delta-live-tables/unity-catalog.html

Thanks, @NandiniN, for your response. The requirement here is not about separating the schema, as we are already handling that with direct publishing mode.

The question is whether we can use a single DLT pipeline to execute multiple groups of tables without causing previously created tables to be dropped from Unity Catalog.

For example, in the first DLT run, tables 1-50 are created in Unity Catalog. However, when we use the same DLT pipeline for another set of tables (51-100), the previously created tables (1-50) are dropped, and only tables 51-100 remain in Unity Catalog. We want to prevent this dropping behavior.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group