Databricks

Tom_Greenwood · 02-01-2024

Hi community,I am using a pyspark udf. The function is being imported from a repo (in the repos section) and registered as a UDF in a the notebook. I am getting a PythonException error when the transformation is run. This is comming from the databric...

Tom_Greenwood · 03-11-2024

No, the wrapper function I showed in the snippet was the only thing that worked but wasn't practical so I've found a work around to not use a udf at all.

Tom_Greenwood · 02-06-2024

Sorry a few mistakes were in my first answer. Here is the corrected version:Thanks for your reply. The function that forms the udf is in repos and the udf is registered and called in a notebook which is not. For most tests I am registering the udf in...

Tom_Greenwood · 02-06-2024

from pyspark.sql.functions import udf from pyspark.sql.types import StringType @udf(returnType=StringType()) def udf_1(*cols): """This works""" def helper_function(code: str) -> str: if code == "spam": return "foo" ...

Tom_Greenwood · 02-06-2024

Thanks for your reply. The function that forms the udf is in repos and the notebook is not. For most tests I am registering the udf in the repo, after importing the function, however I have also tested registering the udf in the file that it's writte...

Databricks

User Stats

User Activity

UDF importing from other modules

Re: UDF importing from other modules

Re: UDF importing from other modules

Re: UDF importing from other modules

Re: UDF importing from other modules