how we can use config file to change pysparks dataframe names without hardcoding

[SailajaB]

Hi,

Can we use config file to change pyspark dataframe attribute names (root, nested of both struct and array type) .

Actually in input we are getting attributes in lowercase we need to convert them into camel case(please note we don't have any separators in attribute name to use or build camel functions or libraries.

We are trying to implement it in pyspark.

Please add your suggestions over here

Thank you

[Kaniz]

Hi @Sailaja B​ ! Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

[Kaniz]

Hi @Sailaja B​ , Can you show your input file if you're comfortable with it?

[Anonymous]

Hi @Sailaja B​ 

Hope you are well.

Just wanted to see if you were able to find an answer to your question and would you be happy to share it with us? It would be really helpful for the other members too.

Thanks!

[SailajaB · Accepted solution]

We have created config file in csv format and converted that into dict key value pairs to chanage the dataframe attribute names.

[Anonymous]

Hi @Sailaja B​ 

This is awesome!

Thanks for coming in and posting the solution. We really appreciate it.

Cheers!