Ashwin_DSA
Databricks Employee
Databricks Employee

Hi @SantiNath_Dey ,

Given you’re dealing with multiple complex JSON structures, I agree with a metadata‑driven approach rather than hard‑coding the flattening logic.

At a high level:

  • Define a small set of config tables:

    • One for patterns... as in which JSON shape/source you’re handling, with a detection rule.
    • One for entities per pattern... which relational tables you want.. for example, customer, orders, order_lines, etc.
    • Perhaps another one for the entity explode paths?
  • Build a generic flattening/orchestration layer that:

    1. Reads the raw JSON.
    2. Uses the pattern config to pick the right pattern.
    3. For each entity in that pattern, follow the configured explode paths (arrays --> child tables) and select the configured columns, including parent keys as FKs.

The initial config could feel a bit heavier, but it scales really well. When a new JSON pattern shows up, you only add rows to the config tables (new pattern, entities, and exploded paths). You don’t touch the flattening/orchestration code at all.

Does that give you a rough idea to start with?

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

Regards,
Ashwin | Delivery Solution Architect @ Databricks
Helping you build and scale the Data Intelligence Platform.
***Opinions are my own***