Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-25-2022 09:13 AM
Hi @Davide Cagnoni. Please see my answer to this post https://community.databricks.com/s/question/0D53f00001mUyh2CAC/limitations-with-udfs-wrapping-module...
I will copy it here for you:
If your notebook is in the same Repo as the module, this should work without any modifications to the sys path.
If your notebook is not in the same Repo as the module, you may need to ensure that the sys path is correct on all nodes in your cluster that need the module. For example, this code should work for you:
# Create a wrapper function around my module that updates the sys path
import sys
def my_wrapper_function(x):
sys.path.append("/Workspace/Repos/user_name/repo_name")
from repo_name import lib_function
return lib_function(x)
# Define the UDF
my_udf = udf(lambda col_name: my_wrapper_function(col_name))
# This should work now
df = df.withColumn(F.col("col1"), my_udf(F.col("col1")))