Hi,
i'm using concat_ws in scala to calculate a checksum for the dataframe, i.e.:
df.withColumn("CHECKSUM", sha2(functions.concat_ws("", dataframe.columns.map(col): _*), 512))
I have one example here with just 24 columns that already throws the following exception: org.apache.spark.sql.AnalysisException: cannot resolve 'concat_ws('', <list of the columns)
Any ideas what's happening? I assume the list get's too long (char wise), but I have no idea how to make this work.
Thanks!