Compiling Flattened Dataframe back to Struct Columns

etsyal1e2r3 — Sun, 04 Jun 2023 00:00:16 GMT

I have a dataframe with this format of columns:

[`first.second.third` , `alpha.bravo.test1` , `alpha.bravo.test2`]

I'd like to get an output dataframe of this:

[               `first`                  |              `alpha`             ]
-------------------------------------------
[{'second':                        |    [{'bravo':
        [{'third': value}]        |         [{'test1': value}     
                                          |         {'test2': value} ]
 }]                                      |    }]

I tried doing something like this but have some errors with using *select_list

AnalysisException: [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column or function parameter with name `F`.`col("_timestamp"),` cannot be resolved.

But manually putting in generated list of select cols with structs works but not exactly what I want.

In the above example the columns were...

[`cloud-provider` , `cloud-region` , `cloud-service-name`]

So the desired result is supposed to be...

[`cloud`]
-------
[{'provider': value},
{'region': value},
{'service': {
        'name': value}}]

Thank you for your help!!!

Re: Compiling Flattened Dataframe back to Struct Columns

etsyal1e2r3 — Tue, 06 Jun 2023 04:43:01 GMT

I have figured out the solution.

topic Re: Compiling Flattened Dataframe back to Struct Columns in Data Engineering

Compiling Flattened Dataframe back to Struct Columns

Re: Compiling Flattened Dataframe back to Struct Columns