cancel
Showing results for 
Search instead for 
Did you mean: 
Warehousing & Analytics
Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Community. Share insights, tips, and best practices for leveraging data for informed decision-making.
cancel
Showing results for 
Search instead for 
Did you mean: 

unable to perform modifications on Table while Using Python UDF in query

doremon11
New Contributor

Here, we're trying to use the Python UDF inside the query.

  • taking the table as function input 
  • converting the table into dataframe 
  • performing modification 
  • converting the dataframe into table 
  • returning the table  

How can we create spark context inside UDF in the  query 

 

 

CREATE FUNCTION fun1(input_table TABLE) RETURNS TABLE
LANGUAGE PYTHON
AS $$
  import pandas as pd
  
  df = spark.sql(f"SELECT * FROM {input_table}")
  def fun(df):
      # Convert table to DataFrame
      df.write.saveAsTable("my_table")
      return my_table
  return fun(input_table)
$$;

 

 

 

 

1 REPLY 1

Kaniz_Fatma
Community Manager
Community Manager

Hi @doremon11Unfortunately, creating a Spark context inside a UDF directly in the query is not possible. The Spark context is a global object and cannot be created within a UDF. UDFs are designed to operate on data within a DataFrame, not to create or manage Spark contexts.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!