transform a dataframe column as concatenated string

[geertvanhove]

Hello,
I have a single column dataframe and I want to transform the content into a string

EG df=

abc

def

xyz

To

abc, def, xyz

Thanks

[Kaniz]

Hi @geertvanhove , You can use the df.tolist() method to convert your dataframe into a list, and then use the ', '.join() method to concatenate the elements of the list into a string. 

Here's an example:

df = pd.DataFrame({'ABC': ['abc', 'def', 'xyz']})
result = ', '.join(df.tolist())
print(result) # Output: abc, def, XYZ

[Kaniz]

Hi @geertvanhove , you should apply the .tolist() function to a specific column or series within your DataFrame, rather than the entire DataFrame.

[geertvanhove]

Thanks, but can you give me an example for this because following still gives me an error: 

result = ', '.join(df.select(col("meterId")).tolist())

[geertvanhove]

I get following error:

'DataFrame' object has no attribute 'tolist'

[Kaniz]

Hi @geertvanhove , Could you paste the code you're trying?

[geertvanhove]

sure:

%python

from pyspark.sql.functions import from_json, col, concat_ws

from pyspark.sql.types import *

schema = StructType([StructField('meterDateTime', StringType(), True), StructField('meterId', LongType(), True), StructField('meteringState', StringType(), True), StructField('value', DoubleType(), True), StructField('versionTimestamp', StringType(), True), StructField('file_name', StringType(), False), StructField('file_modification_time', TimestampType(), False)])

df = ( spark

        .read

        .format("json")

        .schema(schema)

        .load(f'{path_sep}/*/*/*/*.json')

        .select("meterId")

  )

result = ', '.join(df.tolist())

print(result)

[Kaniz]

Hi @geertvanhove , I gave you the code with screenshot.