ModuleNotFoundError when run with foreachBatch on serverless mode

[fury-kata]

I using Notebooks to do some transformations 

I install a new whl:

%pip install --force-reinstall /Workspace/<my_lib>.whl
%restart_python

Then I  successfully import the installed lib

from my_lib.core import test

However when I run my code with foreachBatch it raises ModuleNotFoundError: No module named 'my_lib'.

This is my code:

from my_lib.utils import clogs
logs = clogs.logs()
def _test(df, b):
    logs.add_logs('test')

mystream =  spark.readStream\
                                    .table('my_tbale') \
                                .writeStream\
                                    .format("delta")\
                                    .foreachBatch(_test)\
                                    .trigger(once=True) \
                                .start()
mystream.awaitTermination()
streaming_silver.awaitTermination()

It raises an error: ModuleNotFoundError: No module named 'my_lib'.

Please help

[Kaniz_Fatma]

Hi @fury-kata, 

• Make sure that the path to your custom module is correctly added to the Python path (sys.path). You mentioned installing the .whl file, so ensure that the installation path is accessible from your Databricks notebook.
• Verify that the path to your module is correctly specified. If it’s within a repository, ensure that the repository path is included in sys.path.
• Databricks treats notebooks and Python files differently. Notebooks cannot be imported as Python modules directly. Only Python files can be used as modules.
• Check if your .py file (e.g., my_lib.py) contains the following line at the beginning:

• # Databricks notebook source
  

• If it does, remove this line to treat the file as a regular Python module.
• Sometimes, changes to the Python path require restarting the cluster. After modifying the path, restart your Databricks cluster to apply the changes.
• Ensure that your custom module files have the .py extension.
• Double-check that the file you’re trying to import is indeed a Python file.
• If your custom module relies on external libraries, make sure those libraries are installed in your Databricks cluster.
• You can use notebook-scoped library installation commands in a cell:

  %sh dbutils.library.installPyPI("mlflow")
  dbutils.library.restartPython()
  

  Replace "mlflow" with the actual library name if needed¹.
• If you’re using a specific cluster configuration (e.g., ML cluster), ensure that it includes the necessary libraries and paths.

• Remember to verify each step carefully, and hopefully, one of these solutions will resolve the issue. If you need further assistance, feel free to ask! 

[fury-kata]

Thank @Kaniz_Fatma for your response.

Today, I re-run my job again, without any changes. It doesn’t raise module not found my_lib as I mentioned above, but it raises the Access Denied on my S3 bucket. I don't see anywhere to set my IAMr or instance profile on the serverless as I did with provision compute.