Manipulate Column that is an array of objects

[Raymond_Garcia]

I have a column that is an array of objects, let's call it ARRAY, and now I would like to query / manipulate, the elements object without using explode function, this is an example, for each element in that column I would like to create a path. .withColumn("image_path", concat_ws("", lit(imagePath), lit("/"), $"ARRAY[*].attribute.attribute1", lit("/"), $"ARRAY[*].attribute.attribute2", lit(".jpg")))  I would a column that contains a array of images paths like [imagePath1, imagePath2]. Do you have any suggestion?. Thanks!

[Hubert-Dudek · Accepted solution]

@Raymond Garcia​ , Please just use the Spark transform function. It will get every element from your array, and you can concatenate it with string as in your example. Here is an example of a transform function:

df = spark.createDataFrame([(1, [1, 2, 3, 4])], ("key", "values"))
 
df.select(transform("values", lambda x: x * 2).alias("doubled")).show()
+------------+
|     doubled|
+------------+
|[2, 4, 6, 8]|
+------------+

[Raymond_Garcia]

Thanks! I am going to test it, and I will come back

[Raymond_Garcia]

Hello I am working with Scala, and I used somehing similar:

def play(col: Column): Column = {

 concat_ws("", lit(imagePath), lit("/"), col("field1"), lit("/"), col("field2"), lit(".ext"))

}

val variable = spark.lot_of_stuff.

                 .withColumn("image_path", transform(col("column1.field1"), play(_)))