topic Re: Manipulate Column that is an array of objects in Data Engineering

Manipulate Column that is an array of objects

Raymond_Garcia — Fri, 20 May 2022 22:50:16 GMT

I have a column that is an array of objects, let's call it ARRAY, and now I would like to query / manipulate, the elements object without using explode function, this is an example, for each element in that column I would like to create a path. .withColumn("image_path", concat_ws("", lit(imagePath), lit("/"), $"ARRAY[*].attribute.attribute1", lit("/"), $"ARRAY[*].attribute.attribute2", lit(".jpg"))) I would a column that contains a array of images paths like [imagePath1, imagePath2]. Do you have any suggestion?. Thanks!

Re: Manipulate Column that is an array of objects

Hubert-Dudek — Sat, 21 May 2022 09:52:15 GMT

@Raymond Garcia , Please just use the Spark transform function. It will get every element from your array, and you can concatenate it with string as in your example. Here is an example of a transform function:

df = spark.createDataFrame([(1, [1, 2, 3, 4])], ("key", "values"))
 
df.select(transform("values", lambda x: x * 2).alias("doubled")).show()
+------------+
|     doubled|
+------------+
|[2, 4, 6, 8]|
+------------+

Re: Manipulate Column that is an array of objects

Raymond_Garcia — Mon, 23 May 2022 14:13:29 GMT

Thanks! I am going to test it, and I will come back

Re: Manipulate Column that is an array of objects

Raymond_Garcia — Mon, 23 May 2022 19:29:17 GMT

Hello I am working with Scala, and I used somehing similar:

def play(col: Column): Column = {

concat_ws("", lit(imagePath), lit("/"), col("field1"), lit("/"), col("field2"), lit(".ext"))

}

val variable = spark.lot_of_stuff.

.withColumn("image_path", transform(col("column1.field1"), play(_)))