MetaData Framework for Multi-Level Silver Layer PK/FK Creation

SantiNath_Dey
Contributor II

Hi Team,
Our source data originates from MongoDB as hierarchical JSON and is ingested into our Silver layer, where it is already normalized into tabular structures. However, primary and foreign key constraints are not enforced during ingestion. We must establish and maintain PK/FK relational integrity post-loading through an automated post-processing step.The Challenge:The source system features a deep, multi-level parent-child tree hierarchy (e.g., emp -> emp_addr -> emp_addr_ofc). We require a metadata-driven modeling framework and script execution strategy to dynamically generate surrogate keys, resolve parent-child references top-down, and enforce relational integrity across all levels.

Please find below screen shot for your reference.

SantiNath_Dey_0-1788703344844.png