Problem Statement:
We have a scenario where we get the data from the source in the format of (in actual 20 Levels and number of fields are more than 4 but for ease of understanding letโs consider below)
The actual code involved 20 levels of 4-5 fields each and with 20 joins/CTE for getting outputs.
Question:
Power BI level handling, Path Function is great and can generate the | separated lineage but, while visualizing it needs the separate column creation
But it will be overkill for the render time,
Do we have any similar in-built functions/libraries in Databricks (SQL,Pyspark, Dataframes,pandas) ??
Could you please suggest the optimized approach for this problem statement? (Consider the performance)
Kindly find the example in attachment.