Problem Statement:

We have a scenario where we get the data from the source in the format of (in actual 20 Levels and number of fields are more than 4 but for ease of understanding let’s consider below)

The actual code involved 20 levels of 4-5 fields each and with 20 joins/CTE for getting outputs.

Question:

Power BI level handling, Path Function is great and can generate the | separated lineage but, while visualizing it needs the separate column creation

But it will be overkill for the render time,

Do we have any similar in-built functions/libraries in Databricks (SQL,Pyspark, Dataframes,pandas) ??

Could you please suggest the optimized approach for this problem statement? (Consider the performance)

Kindly find the example in attachment.