Call python image function in pyspark

[MichaelO]

I have a function for rotating images written in python:

from PIL import Image
def rotate_image(image, rotation_angle):
  im = Image.open(image)
  out = im.rotate(rotation_angle, expand = True)
  return out

I now want to use this function as a pyspark udf.

import pyspark.sql.functions as fn
 
df = spark.read.format("image").load("image2.jpg")
#Trying to register the function here
#Not sure what data type to declare the image as here
_udf = fn.udf(rotate_image) 
#To apply the function - not really sure what I am doing here
df.withColumn('x1', _udf(array('image', 90))).show() 

Can you please assist with registering and using udf with images.

[FabriceDeseyn]

Hi

Images are loaded in 1 Struct column containing multiple fields according to image data source docs.

I guess the return datatype can be whatever you want it to be as long as it matches the output of the UDF.

[artsheiko · Accepted solution]

Let's try the following :

from pyspark.ml.image import ImageSchema, ImageTransformer
from pyspark.sql.functions import udf
from pyspark.sql.types import StringType
 
angle = 90
 
def rotate_image(image):
    np_array = ImageSchema.toNDArray(image)
    rotated = np.rot90(np_array, k=angle//90, axes=(1, 0))
    return ImageSchema.fromNDArray(rotated)
 
rotate_udf = udf(rotate_image, StringType())
 
image_df = spark.read.format("image").load("path/to/image/directory")
rotated_images_df = ImageTransformer().setOutputCol("rotated").transform(image_df).withColumn("rotated", rotate_udf("image"))

[Anonymous]

Hi @Michael Okelola​ 

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance! 

[Raluka]

Stock photos, I've come to realize, are the catalysts of imagination. This website's vast reservoir of images new york seal sparks ideas that ripple through my projects. They empower me to envision the previously unimagined, helping me breathe life into concepts and messages that captivate and inspire.