cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Manipulate Column that is an array of objects

Raymond_Garcia
Contributor II

I have a column that is an array of objects, let's call it ARRAY, and now I would like to query / manipulate, the elements object without using explode function, this is an example, for each element in that column I would like to create a path. .withColumn("image_path", concat_ws("", lit(imagePath), lit("/"), $"ARRAY[*].attribute.attribute1", lit("/"), $"ARRAY[*].attribute.attribute2", lit(".jpg")))  I would a column that contains a array of images paths like [imagePath1, imagePath2]. Do you have any suggestion?. Thanks!

1 ACCEPTED SOLUTION

Accepted Solutions

Hubert-Dudek
Esteemed Contributor III

@Raymond Garcia​ , Please just use the Spark transform function. It will get every element from your array, and you can concatenate it with string as in your example. Here is an example of a transform function:

df = spark.createDataFrame([(1, [1, 2, 3, 4])], ("key", "values"))
 
df.select(transform("values", lambda x: x * 2).alias("doubled")).show()
+------------+
|     doubled|
+------------+
|[2, 4, 6, 8]|
+------------+

View solution in original post

3 REPLIES 3

Hubert-Dudek
Esteemed Contributor III

@Raymond Garcia​ , Please just use the Spark transform function. It will get every element from your array, and you can concatenate it with string as in your example. Here is an example of a transform function:

df = spark.createDataFrame([(1, [1, 2, 3, 4])], ("key", "values"))
 
df.select(transform("values", lambda x: x * 2).alias("doubled")).show()
+------------+
|     doubled|
+------------+
|[2, 4, 6, 8]|
+------------+

Thanks! I am going to test it, and I will come back

Raymond_Garcia
Contributor II

Hello I am working with Scala, and I used somehing similar:

def play(col: Column): Column = {

 concat_ws("", lit(imagePath), lit("/"), col("field1"), lit("/"), col("field2"), lit(".ext"))

}

val variable = spark.lot_of_stuff.

                 .withColumn("image_path", transform(col("column1.field1"), play(_)))