Problems with extending DataFrame-object with custom method in DBR 14.0

[hanspetter]

I noticed that extending the DataFrame object with a custom method does not work in DBR 14.0 This works fine in when using DBR 13.3. The below code throws a "TypeError: 'Column' object is not callable".

from pyspark.sql.functions import col,lit
from pyspark.sql.dataframe import DataFrame

def custom_method(df):
  new_df = df.withColumn("new_column", lit(1))
  return new_df

setattr(DataFrame, "custom_method", custom_method)
#Also tried: DataFrame.custom_method = custom_method

d = [{'empid': 1, 'name': 'Test'}]

my_df = spark.createDataFrame(d)

my_df = my_df.custom_method()

 Is this expected behaviour?

[Kaniz]

Hi @hanspetter, It's not explicitly stated that extending the DataFrame object with a custom method is not supported in 14.0. However, we do know that there are limitations to the PySpark DataFrame API and Databricks Connect, which could potentially affect this.

The error you're experiencing, "TypeError:’ Column’ object is not callable", is typically thrown when you're trying to call a function on an object that's not a function. This could be due to the differences between 14.0 and 13.3, but it's hard to say for certain without more context or information.