Importing python module

tariq
New Contributor III

I'm not sure how a simple thing like importing a module in python can be so broken in such a product. First, I was able to make it work using the following:

import sys
sys.path.append("/Workspace/Repos/Github Repo/sparkling-to-databricks/src")
from utils.some_util import *

I was able to use the imported function. But then I restarted the cluster and this would not work even though the path is in sys.path.

I also tried the following:

spark.sparkContext.addPyFile("/Workspace/Repos/Github Repo/my-repo/src/utils/some_util.py")

This did not work either. Can someone please tell me what I'm doing wrong here and suggest a solution. Thanks.