topic Re: Why does my Notebook fails if I try to load a function from another Notebook in Repos in Databricks in Data Engineering

Why does my Notebook fails if I try to load a function from another Notebook in Repos in Databricks

shubhamb — Wed, 01 Jun 2022 08:26:45 GMT

My function in func.py

def lower_events(df):
 return df.withColumn("event",f.lower(f.col("event")))

My main notebook

import pyspark.sql.functions as f
from pyspark.sql.functions import udf, col, lower
import sys
 
sys.path.append("..")
from folder.func.  import   lower_events
#from folder  import   func
 
df_clean = lower_events(df)

I am getting this error

NameError: name 'f' is not defined

Re: Why does my Notebook fails if I try to load a function from another Notebook in Repos in Databricks

-werners- — Wed, 01 Jun 2022 12:25:11 GMT

@Shubham Biswas, so func.py is a plain Python file, not a Databricks notebook?

Can you also check the following topic?

https://community.databricks.com/s/question/0D53f00001mUyh2CAC/limitations-with-udfs-wrapping-modules-imported-via-repos-files

Re: Why does my Notebook fails if I try to load a function from another Notebook in Repos in Databricks

shubhamb — Thu, 16 Jun 2022 05:23:59 GMT

@Kaniz Fatma https://community.databricks.com/s/question/0D58Y00008ouo6xSAA/how-to-fetch-environmental-variables-saved-in-one-notebook-into-another-notebook-in-databricks-repos-and-notebooks

Can you please look into this