Strange Error with custom module in delta live table pipeline

[gabe123]

The chunk of code in question

sys.path.append(
    spark.conf.get("util_path", "/Workspace/Repos/Production/loch-ness/utils/")
)
from broker_utils import extract_day_with_suffix, proper_case_address_udf, proper_case_last_name_first_udf, proper_case_udf

Encountering the ModuleNotFoundError: No module named 'broker_utils' on only the broker_utils import. We are importing other .py custom utility files and not encountering the error. We are not encountering the error importing broker_utils in other pipeline jobs nor is it in an issue in notebooks.

Any help is appreciated, thanks.

[Kaniz]

Hi @gabe123 , It seems like you’re encountering a ModuleNotFoundError when trying to import the broker_utils module in your Python code.

Let’s troubleshoot this issue step by step:

1. Check Module Location:

   • First, ensure that the broker_utils.py file exists in the specified path: /Workspace/Repos/Production/loch-ness/utils/. Verify that the file name is correct and that it contains the necessary functions or classes.
   • Confirm that the file is accessible and readable by the Python interpreter.

2. Check Python Environment:

   • Make sure you are using the same Python environment (virtual environment or system-wide Python) where other custom utility files are successfully imported.
   • Sometimes, different environments can lead to module import issues.

3. Check sys.path:

   • The sys.path.append(...) line adds a directory to the Python search path. Ensure that the specified path is correct and points to the directory containing broker_utils.py.
   • You can print sys.path to verify that the path has been added correctly.

4. Check Module Name:

   • Double-check that the module name is spelled correctly. Python is case-sensitive, so make sure it matches the actual filename (broker_utils.py).
   • Verify that there are no typos or extra spaces in the import statement.

5. Circular Dependencies:

   • Check if there are any circular dependencies between modules. For example, if broker_utils imports another module that also imports broker_utils, it can cause issues.
   • Review the code in broker_utils.py and any other modules it imports.
   •  

6. Import Order:

   • The order of imports matters. Ensure that the sys.path.append(...) line is executed before the import broker_utils statement.
   • If other custom utility files are imported before broker_utils, make sure their order is correct.

7. Check for Typos or Syntax Errors:

   • Look for any typos or syntax errors in the broker_utils.py file. Even a small mistake can prevent successful imports.

8. Check for Hidden Characters:

   • Sometimes hidden characters (such as invisible whitespace) can cause issues. Open the broker_utils.py file in a text editor and check for any unusual characters.

9. Verify File Permissions:

   • Ensure that the file permissions allow reading and execution. If the file is not readable, Python won’t be able to import it.

10. Test in a Minimal Environment:

    • Create a minimal test script that only imports broker_utils and nothing else. Run it outside of your main project to isolate the issue.
    • This will help identify whether the problem is specific to your project or a broader issue.

Remember to check each of these steps carefully, and hopefully, you’ll find the root cause of the ModuleNotFoundError. If you continue to face issues, feel free to provide more details, and we can explore further solutions! 😊🚀