- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-26-2024 01:13 PM - edited 08-26-2024 01:29 PM
Hi Dave,
You can solve this by putting your utils into a python file and referencing your .py file in the DLT notebook. I provided a template for the python file below:
STEP 1:
#import functions
from pyspark.sql import SparkSession
import IPython
dbutils = IPython.get_ipython().user_ns["dbutils"]
spark = SparkSession.builder.getOrCreate()
def myfunc1():
test = 1
STEP 2: You will need to create a __init__.py file in the same directory your utils.py file lives.
STEP 3:
In your DLT notebook, you'll need to append your sys path and then import your utils file as a library.
# set path
import sys
sys.path.append("/Workspace/utils_folder")
# import libraries
import dlt
import my_utils
I suggest to avoid naming your package with existing packages, e,g; pandas as a file name. I also suggest you put your utils file in a separate path from all your other files. This will make appending your path less risky.