We are implementing a data ingestion framework where data flows from a foreign catalog (source) to a raw layer (Delta tables) and then to a bronze layer (DLT streaming tables). Currently, each Business Unit (BU) has a separate workflow and DLT pipeline (e.g., BU1 → Workflow1 & DLT1, BU2 → Workflow2 & DLT2).

To improve efficiency, we aim to create a generic workflow and DLT pipeline that supports multiple BUs without maintaining separate workflows. While workflow-level adjustments are feasible, the challenge arises at the DLT pipeline level:

• If we modify the path where table metadata is stored, tables get dropped from the catalog, impacting the pipeline.
• We need a way to segregate BU-specific tables within the same DLT pipeline while ensuring they persist across updates.