Updated sys.path not working any more

Liliana — Wed, 25 Oct 2023 19:23:58 GMT

We have a monorepo so our pyspark notebooks do not use namespace relative to the root of the repo. Thus the default sys.path of repo root and cwd does not work. We used to package a whl dependency but recently moved to having code update sys.path with sys.path.append(). While testing a while ago, this seemed to work, this week, it seems updated sys.path no longer works even on a simple driver only / personal cluster. We are running DBR 13.3. Has anyone else run into this or have suggestions on what solution may work? Thank you!

Re: Updated sys.path not working any more

jose_gonzalez — Wed, 01 Nov 2023 17:27:19 GMT

Hi @Liliana ,

Just a friendly follow-up. Have you had a chance to review my colleague's response to your inquiry? Did it prove helpful, or are you still in need of assistance? Your response would be greatly appreciated.

topic Updated sys.path not working any more in Data Engineering

Updated sys.path not working any more

Re: Updated sys.path not working any more