unable to perform modifications on Table while Using Python UDF in query

[doremon11]

Here, we're trying to use the Python UDF inside the query.

• taking the table as function input 
• converting the table into dataframe 
• performing modification 
• converting the dataframe into table 
• returning the table  

How can we create spark context inside UDF in the  query 

CREATE FUNCTION fun1(input_table TABLE) RETURNS TABLE
LANGUAGE PYTHON
AS $$
  import pandas as pd
  
  df = spark.sql(f"SELECT * FROM {input_table}")
  def fun(df):
      # Convert table to DataFrame
      df.write.saveAsTable("my_table")
      return my_table
  return fun(input_table)
$$;

[Kaniz]

Hi @doremon11, Unfortunately, creating a Spark context inside a UDF directly in the query is not possible. The Spark context is a global object and cannot be created within a UDF. UDFs are designed to operate on data within a DataFrame, not to create or manage Spark contexts.