https://community.databricks.com/t5/data-engineering/problems-with-extending-dataframe-object-with-custom-method-in/m-p/45656#M27963 <P>I noticed that extending the DataFrame object with a custom method does not work in DBR 14.0 This works fine in when using DBR 13.3. The below code throws a "<SPAN class="">TypeError</SPAN><SPAN>: 'Column' object is not callable</SPAN>".</P><P>&nbsp;</P><P>&nbsp;</P><LI-CODE lang="python">from pyspark.sql.functions import col,lit from pyspark.sql.dataframe import DataFrame def custom_method(df): new_df = df.withColumn("new_column", lit(1)) return new_df setattr(DataFrame, "custom_method", custom_method) #Also tried: DataFrame.custom_method = custom_method d = [{'empid': 1, 'name': 'Test'}] my_df = spark.createDataFrame(d) my_df = my_df.custom_method()</LI-CODE><P>&nbsp;</P><P>&nbsp;</P><P>&nbsp;Is this expected behaviour?</P> Fri, 22 Sep 2023 12:12:27 GMT hanspetter 2023-09-22T12:12:27Z https://community.databricks.com/t5/data-engineering/problems-with-extending-dataframe-object-with-custom-method-in/m-p/45656#M27963 <P>I noticed that extending the DataFrame object with a custom method does not work in DBR 14.0 This works fine in when using DBR 13.3. The below code throws a "<SPAN class="">TypeError</SPAN><SPAN>: 'Column' object is not callable</SPAN>".</P><P>&nbsp;</P><P>&nbsp;</P><LI-CODE lang="python">from pyspark.sql.functions import col,lit from pyspark.sql.dataframe import DataFrame def custom_method(df): new_df = df.withColumn("new_column", lit(1)) return new_df setattr(DataFrame, "custom_method", custom_method) #Also tried: DataFrame.custom_method = custom_method d = [{'empid': 1, 'name': 'Test'}] my_df = spark.createDataFrame(d) my_df = my_df.custom_method()</LI-CODE><P>&nbsp;</P><P>&nbsp;</P><P>&nbsp;Is this expected behaviour?</P> Fri, 22 Sep 2023 12:12:27 GMT https://community.databricks.com/t5/data-engineering/problems-with-extending-dataframe-object-with-custom-method-in/m-p/45656#M27963 hanspetter 2023-09-22T12:12:27Z