unable to perform modifications on Table while Using Python UDF in query

doremon11 — Thu, 07 Mar 2024 11:36:08 GMT

Here, we're trying to use the Python UDF inside the query.

taking the table as function input
converting the table into dataframe
performing modification
converting the dataframe into table
returning the table

How can we create spark context inside UDF in the query

CREATE FUNCTION fun1(input_table TABLE) RETURNS TABLE LANGUAGE PYTHON AS $$ import pandas as pd df = spark.sql(f"SELECT * FROM {input_table}") def fun(df): # Convert table to DataFrame df.write.saveAsTable("my_table") return my_table return fun(input_table) $$;

Re: unable to perform modifications on Table while Using Python UDF in query

Vidhi_Khaitan — Wed, 04 Jun 2025 04:20:00 GMT

Hi team,
I believe you cannot create or access a SparkSession or run Spark operations like spark.sql() directly inside a Python UDF. input_table is a table argument, not a string with a table name. You receive it as a pandas DataFrame when using RETURNS TABLE

You need to define your logic outside SQL in a notebook and use regular Spark APIs:

Then call process_table("my_table") in your notebook or job. Hope this helps!

topic unable to perform modifications on Table while Using Python UDF in query in Warehousing & Analytics

unable to perform modifications on Table while Using Python UDF in query

Re: unable to perform modifications on Table while Using Python UDF in query