kenny_hero
New Contributor III

@Hubert-Dudek, thank you for your response. I really appreciate it.

However, I still cannot get the import to work even after I follow your instructions. Here is the folder structure:

import_test_project_structure.png

The transformer code is below:

from pyspark import pipelines as dp

from project.utils.logger import get_logger
#from utils.logger import get_logger

@dp.table
def sample_users():
    logger = get_logger(__name__)
    logger.info('Running sample_users transformer')
    return (
        spark.read.table("samples.wanderbricks.users")
        .select("user_id", "email", "name", "user_type")
    )

After deploying the bundle, I tried running the job but it still fails to import the logger module.

Traceback (most recent call last):
File "/Workspace/Users/.../.bundle/import_test/dev/files/src/project/pipelines/import_test/transformations/sample_users.py", cell 1, line 3
      1 from pyspark import pipelines as dp
----> 3 from project.utils.logger import get_logger
      4 #from utils.logger import get_logger
      6 @dp.table
      7 def sample_users():

File "/databricks/python_shell/lib/dbruntime/autoreload/discoverability/hook.py", line 71, in AutoreloadDiscoverabilityHook._patched_import(self, name, *args, **kwargs)
     65 if not self._should_hint and (
     66     (module := sys.modules.get(absolute_name)) is not None and
     67     (fname := get_allowed_file_name_or_none(module)) is not None and
     68     (mtime := os.stat(fname).st_mtime) > self.last_mtime_by_modname.get(
     69         absolute_name, float("inf")) and not self._should_hint):
     70     self._should_hint = True
---> 71 module = self._original_builtins_import(name, *args, **kwargs)
     72 if (fname := fname or get_allowed_file_name_or_none(module)) is not None:
     73     mtime = mtime or os.stat(fname).st_mtime

ModuleNotFoundError: No module named 'project'
Error: [update_progress]  Update 803be7 is FAILED.

 I tried importing from both "project.utils.logger" and "utils.logger" and neither worked.

Can you help me figure out what I'm doing wrong?