how we can use config file to change pysparks dataframe names without hardcoding

SailajaB
Databricks Partner

‎02-24-2022 05:16 AM

Hi,

Can we use config file to change pyspark dataframe attribute names (root, nested of both struct and array type) .

Actually in input we are getting attributes in lowercase we need to convert them into camel case(please note we don't have any separators in attribute name to use or build camel functions or libraries.

We are trying to implement it in pyspark.

Please add your suggestions over here

Thank you

Labels:

Anonymous
Not applicable

‎04-28-2022 09:53 AM

Hi @Sailaja B​ 

Hope you are well.

Just wanted to see if you were able to find an answer to your question and would you be happy to share it with us? It would be really helpful for the other members too.

Thanks!

SailajaB
Databricks Partner
In response to Anonymous

‎04-29-2022 05:04 AM

We have created config file in csv format and converted that into dict key value pairs to chanage the dataframe attribute names.

View solution in original post

Anonymous
Not applicable

‎04-29-2022 07:56 AM

Hi @Sailaja B​ 

This is awesome!

Thanks for coming in and posting the solution. We really appreciate it.

Cheers!