cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Updated sys.path not working any more

Liliana
New Contributor

We have a monorepo so our pyspark notebooks do not use namespace relative to the root of the repo. Thus the default sys.path of repo root and cwd does not work. We used to package a whl dependency but recently moved to having code update sys.path with sys.path.append(). While testing a while ago, this seemed to work, this week, it seems updated sys.path no longer works even on a simple driver only / personal cluster. We are running DBR 13.3. Has anyone else run into this or have suggestions on what solution may work? Thank you!

2 REPLIES 2

Kaniz_Fatma
Community Manager
Community Manager

Hi @Liliana , If updating sys.path using sys.path.append() is not working in your PySpark notebooks, you can try the following alternative solutions:

  1. Use the PYTHONPATH environment variable:

Set the PYTHONPATH environment variable in your cluster to include the path to the root of your monorepo to import modules relative to the root of your repo in your PySpark notebooks.

You can set the PYTHONPATH environment variable in SparkConf().set() method or using a configuration file.

  1. Add the root of your monorepo to the zip file distributed across your nodes:

Package your entire monorepo into a zip file and distribute it to all nodes of your cluster using --py-files option when submitting PySpark notebooks as jobs to the cluster.

By using one of these alternative methods, you can import modules relative to the root of your monorepo in your PySpark notebooks without any issues, even if updating sys.path using sys.path.append() is not working.

jose_gonzalez
Moderator
Moderator

 Hi @Liliana ,

Just a friendly follow-up. Have you had a chance to review my colleague's response to your inquiry? Did it prove helpful, or are you still in need of assistance? Your response would be greatly appreciated.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!