How to flatten a nested recursive JSON struct to a list of struct

Wayne — Thu, 30 Nov 2023 21:34:16 GMT

This is from Spark Event log on Event SparkListenerSQLExecutionStart.
How to flatten the sparkPlanInfo struct into an array of the same struct, then later explode it. Note that the element children is an array containing the parent struct, and the level of nesting could be 0 to any random number.

topic How to flatten a nested recursive JSON struct to a list of struct in Data Engineering

How to flatten a nested recursive JSON struct to a list of struct