topic Call python image function in pyspark in Data Engineering

Call python image function in pyspark

MichaelO — Fri, 05 May 2023 09:07:40 GMT

I have a function for rotating images written in python:

from PIL import Image
def rotate_image(image, rotation_angle):
  im = Image.open(image)
  out = im.rotate(rotation_angle, expand = True)
  return out

I now want to use this function as a pyspark udf.

import pyspark.sql.functions as fn
 
df = spark.read.format("image").load("image2.jpg")
#Trying to register the function here
#Not sure what data type to declare the image as here
_udf = fn.udf(rotate_image) 
#To apply the function - not really sure what I am doing here
df.withColumn('x1', _udf(array('image', 90))).show()

Can you please assist with registering and using udf with images.

Re: Call python image function in pyspark

FabriceDeseyn — Wed, 10 May 2023 15:02:42 GMT

Images are loaded in 1 Struct column containing multiple fields according to image data source docs.

I guess the return datatype can be whatever you want it to be as long as it matches the output of the UDF.

Re: Call python image function in pyspark

artsheiko — Wed, 10 May 2023 15:06:21 GMT

Let's try the following :

from pyspark.ml.image import ImageSchema, ImageTransformer
from pyspark.sql.functions import udf
from pyspark.sql.types import StringType
 
angle = 90
 
def rotate_image(image):
    np_array = ImageSchema.toNDArray(image)
    rotated = np.rot90(np_array, k=angle//90, axes=(1, 0))
    return ImageSchema.fromNDArray(rotated)
 
rotate_udf = udf(rotate_image, StringType())
 
image_df = spark.read.format("image").load("path/to/image/directory")
rotated_images_df = ImageTransformer().setOutputCol("rotated").transform(image_df).withColumn("rotated", rotate_udf("image"))

Re: Call python image function in pyspark

Anonymous — Fri, 19 May 2023 08:17:07 GMT

Hi @Michael Okelola

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!

Re: Call python image function in pyspark

Raluka — Sun, 24 Sep 2023 03:42:04 GMT

Stock photos, I've come to realize, are the catalysts of imagination. This website's vast reservoir of images new york seal sparks ideas that ripple through my projects. They empower me to envision the previously unimagined, helping me breathe life into concepts and messages that captivate and inspire.