- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-11-2026 02:11 AM
Hi @SantiNath_Dey ,
Given you’re dealing with multiple complex JSON structures, I agree with a metadata‑driven approach rather than hard‑coding the flattening logic.
At a high level:
-
Define a small set of config tables:
- One for patterns... as in which JSON shape/source you’re handling, with a detection rule.
- One for entities per pattern... which relational tables you want.. for example, customer, orders, order_lines, etc.
- Perhaps another one for the entity explode paths?
-
Build a generic flattening/orchestration layer that:
- Reads the raw JSON.
- Uses the pattern config to pick the right pattern.
- For each entity in that pattern, follow the configured explode paths (arrays --> child tables) and select the configured columns, including parent keys as FKs.
The initial config could feel a bit heavier, but it scales really well. When a new JSON pattern shows up, you only add rows to the config tables (new pattern, entities, and exploded paths). You don’t touch the flattening/orchestration code at all.
Does that give you a rough idea to start with?
If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.
Ashwin | Delivery Solution Architect @ Databricks
Helping you build and scale the Data Intelligence Platform.
***Opinions are my own***