topic how we can use config file to change pysparks dataframe names without hardcoding in Data Engineering

how we can use config file to change pysparks dataframe names without hardcoding

SailajaB — Thu, 24 Feb 2022 13:16:11 GMT

Hi,

Can we use config file to change pyspark dataframe attribute names (root, nested of both struct and array type) .

Actually in input we are getting attributes in lowercase we need to convert them into camel case(please note we don't have any separators in attribute name to use or build camel functions or libraries.

We are trying to implement it in pyspark.

Please add your suggestions over here

Thank you

Re: how we can use config file to change pysparks dataframe names without hardcoding

Anonymous — Thu, 28 Apr 2022 16:53:17 GMT

Hi @Sailaja B

Hope you are well.

Just wanted to see if you were able to find an answer to your question and would you be happy to share it with us? It would be really helpful for the other members too.

Thanks!

Re: how we can use config file to change pysparks dataframe names without hardcoding

SailajaB — Fri, 29 Apr 2022 12:04:49 GMT

We have created config file in csv format and converted that into dict key value pairs to chanage the dataframe attribute names.

Re: how we can use config file to change pysparks dataframe names without hardcoding

Anonymous — Fri, 29 Apr 2022 14:56:34 GMT

Hi @Sailaja B

This is awesome!

Thanks for coming in and posting the solution. We really appreciate it.

Cheers!