Manipulate Column that is an array of objects

Raymond_Garcia
Contributor II

‎05-20-2022 03:50 PM

I have a column that is an array of objects, let's call it ARRAY, and now I would like to query / manipulate, the elements object without using explode function, this is an example, for each element in that column I would like to create a path. .withColumn("image_path", concat_ws("", lit(imagePath), lit("/"), $"ARRAY[*].attribute.attribute1", lit("/"), $"ARRAY[*].attribute.attribute2", lit(".jpg")))  I would a column that contains a array of images paths like [imagePath1, imagePath2]. Do you have any suggestion?. Thanks!

Hubert-Dudek
Databricks MVP

‎05-21-2022 02:52 AM

@Raymond Garcia​ , Please just use the Spark transform function. It will get every element from your array, and you can concatenate it with string as in your example. Here is an example of a transform function:

df = spark.createDataFrame([(1, [1, 2, 3, 4])], ("key", "values"))
 
df.select(transform("values", lambda x: x * 2).alias("doubled")).show()
+------------+
|     doubled|
+------------+
|[2, 4, 6, 8]|
+------------+


My blog: https://databrickster.medium.com/

View solution in original post

Raymond_Garcia
Contributor II
In response to Hubert-Dudek

‎05-23-2022 07:13 AM

Thanks! I am going to test it, and I will come back

0 Kudos

Raymond_Garcia
Contributor II

‎05-23-2022 12:29 PM

Hello I am working with Scala, and I used somehing similar:

def play(col: Column): Column = {

 concat_ws("", lit(imagePath), lit("/"), col("field1"), lit("/"), col("field2"), lit(".ext"))

}

val variable = spark.lot_of_stuff.

                 .withColumn("image_path", transform(col("column1.field1"), play(_)))