cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Compiling Flattened Dataframe back to Struct Columns

etsyal1e2r3
Honored Contributor

I have a dataframe with this format of columns:

[`first.second.third` , `alpha.bravo.test1` , `alpha.bravo.test2`]

I'd like to get an output dataframe of this:

[               `first`                  |              `alpha`             ]
-------------------------------------------
[{'second':                        |    [{'bravo':
        [{'third': value}]        |         [{'test1': value}     
                                          |         {'test2': value} ]
 }]                                      |    }]

I tried doing something like this but have some errors with using *select_list

AnalysisException: [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column or function parameter with name `F`.`col("_timestamp"),` cannot be resolved.

But manually putting in generated list of select cols with structs works but not exactly what I want.

imageIn the above example the columns were...

[`cloud-provider` , `cloud-region` , `cloud-service-name`]

So the desired result is supposed to be...

[`cloud`]
-------
[{'provider': value},
{'region': value},
{'service': {
        'name': value}}]

Thank you for your help!!!

1 ACCEPTED SOLUTION

Accepted Solutions

etsyal1e2r3
Honored Contributor

I have figured out the solution.

View solution in original post

2 REPLIES 2

etsyal1e2r3
Honored Contributor

I have figured out the solution.

Kaniz
Community Manager
Community Manager

Hi @Tyler Retzlaff​ , Would you please share the solution with our community?

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.