Strange Error with custom module in delta live table pipeline

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.

The chunk of code in question

sys.path.append(
    spark.conf.get("util_path", "/Workspace/Repos/Production/loch-ness/utils/")
)
from broker_utils import extract_day_with_suffix, proper_case_address_udf, proper_case_last_name_first_udf, proper_case_udf

Encountering the ModuleNotFoundError: No module named 'broker_utils' on only the broker_utils import. We are importing other .py custom utility files and not encountering the error. We are not encountering the error importing broker_utils in other pipeline jobs nor is it in an issue in notebooks.

Any help is appreciated, thanks.