topic transform a dataframe column as concatenated string in Data Engineering

transform a dataframe column as concatenated string

geertvanhove — Fri, 17 Nov 2023 16:48:25 GMT

Hello,
I have a single column dataframe and I want to transform the content into a string

EG df=

abc

def

xyz

abc, def, xyz

Thanks

Re: transform a dataframe column as concatenated string

geertvanhove — Thu, 23 Nov 2023 14:00:47 GMT

I get following error:

'DataFrame' object has no attribute 'tolist'

Re: transform a dataframe column as concatenated string

geertvanhove — Thu, 23 Nov 2023 14:24:30 GMT

sure:

%python

from pyspark.sql.functions import from_json, col, concat_ws

from pyspark.sql.types import *

schema = StructType([StructField('meterDateTime', StringType(), True), StructField('meterId', LongType(), True), StructField('meteringState', StringType(), True), StructField('value', DoubleType(), True), StructField('versionTimestamp', StringType(), True), StructField('file_name', StringType(), False), StructField('file_modification_time', TimestampType(), False)])

df = ( spark

.read

.format("json")

.schema(schema)

.load(f'{path_sep}/*/*/*/*.json')

.select("meterId")

)

result = ', '.join(df.tolist())

print(result)

Re: transform a dataframe column as concatenated string

geertvanhove — Thu, 23 Nov 2023 15:01:05 GMT

Thanks, but can you give me an example for this because following still gives me an error:

result = ', '.join(df.select(col("meterId")).tolist())