topic Issue with Generic DLT Pipeline Handling Multiple BUs in Data Engineering

Issue with Generic DLT Pipeline Handling Multiple BUs

RamanBajetha — Thu, 30 Jan 2025 12:02:42 GMT

We are implementing a data ingestion framework where data flows from a foreign catalog (source) to a raw layer (Delta tables) and then to a bronze layer (DLT streaming tables). Currently, each Business Unit (BU) has a separate workflow and DLT pipeline (e.g., BU1 → Workflow1 & DLT1, BU2 → Workflow2 & DLT2).

To improve efficiency, we aim to create a generic workflow and DLT pipeline that supports multiple BUs without maintaining separate workflows. While workflow-level adjustments are feasible, the challenge arises at the DLT pipeline level:

If we modify the path where table metadata is stored, tables get dropped from the catalog, impacting the pipeline.
We need a way to segregate BU-specific tables within the same DLT pipeline while ensuring they persist across updates.

Re: Issue with Generic DLT Pipeline Handling Multiple BUs

NandiniN — Sat, 01 Feb 2025 04:45:24 GMT

You can create separate schemas within the same catalog for each BU. For example, you can have schemas like BU1_schema, BU2_schema, etc., within the same catalog.

By using Unity Catalog, you can segregate BU-specific tables within the same DLT pipeline

https://docs.databricks.com/en/delta-live-tables/unity-catalog.html

Re: Issue with Generic DLT Pipeline Handling Multiple BUs

RamanBajetha — Mon, 03 Feb 2025 06:01:35 GMT

Thanks, @NandiniN, for your response. The requirement here is not about separating the schema, as we are already handling that with direct publishing mode.

The question is whether we can use a single DLT pipeline to execute multiple groups of tables without causing previously created tables to be dropped from Unity Catalog.

For example, in the first DLT run, tables 1-50 are created in Unity Catalog. However, when we use the same DLT pipeline for another set of tables (51-100), the previously created tables (1-50) are dropped, and only tables 51-100 remain in Unity Catalog. We want to prevent this dropping behavior.