Use of venv pack
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-30-2022 04:58 AM
Hi, I very new so this probably sounds stupid...
I'm following the blog on How to Manage Python Dependencies in PySpark:
https://www.databricks.com/blog/2020/12/22/how-to-manage-python-dependencies-in-pyspark.html
...but when I try the packing works fine but I recieve a file not found error (see screenshot). What can I do to fix this?
Thanks in advance.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-30-2022 01:16 PM
There are no stupid questions, only ones not asking them 🙂
What package are you need on the cluster? Maybe it can be installed using ready out-of.the-box options.
My blog: https://databrickster.medium.com/
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-30-2022 09:54 PM
Can you try using an absolute path instead of a relative path for the same?
For example: https://stackoverflow.com/questions/38661464/filenotfounderror-winerror-3
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-30-2022 10:28 PM
Thanks for your answer, I tried but it didn't make any differerence (see attached). I will look again at my environment settings as as the error is coming back with '\\' maybe I just need to delete a '\' somewhere?
