Following best practice, we want to avoid reusing code by putting commonly used transformations into function libraries and then importing and calling those functions where required.
We also want to follow Databricks recommendations to use serverless compute and DLT pipelines. However, I'm unable to successfully find a way to import our function libraries.
I’m following the instructions here:
https://docs.databricks.com/aws/en/dlt/import-workspace-files
I’ve created a file called time_and_date_funclib.py to store commonly used date and time transforms.
According to the docco linked above, I should be able to import the functions to my working notebook using the code:
from time_and_date_funclib import *
However, I get a ModuleNotFoundError:
ModuleNotFoundError: Traceback (most recent call last):
File "/Repos/tam.duran@flickelectric.co.nz/data_lakehouse/Data Engineering/30_silver_to_gold/consumption_data/network_banded_usage", cell 5, line 7
4 import sys, os
5 # sys.path.append(os.path.abspath("/Workspace/data_lakehouse/Data Engineering/00_ingestion/lib/time_and_date_funclib"))
----> 7 from time_and_date_funclib import *
ModuleNotFoundError: No module named 'time_and_date_funclib'
I get this error even if time_and_date_funclib.py is stored in the same location as the calling notebook (in case I was formatting the absolute path incorrectly). I have also tried storing it at the root of the Git folder, per the note below (copied from the docco linked above).
note
If your notebook imports modules or packages from a workspace files path or a Git folders path different from the notebook directory, you must manually append the path to the files using sys.path.append().
If you are importing a file from a Git folder, you must prepend /Workspace/ to the path. For example, sys.path.append('/Workspace/...'). Omitting /Workspace/ from the path results in an error.
If the modules or packages are stored in the same directory as the notebook, you do not need to append the path manually. You also do not need to manually append the path when importing from the root directory of a Git folder because the root directory is automatically appended to the path.
Has anyone got this working? What am I missing here?
