topic Use of venv pack in Data Engineering

Use of venv pack

c038644 — Tue, 30 Aug 2022 11:58:07 GMT

Hi, I very new so this probably sounds stupid...

I'm following the blog on How to Manage Python Dependencies in PySpark:

https://www.databricks.com/blog/2020/12/22/how-to-manage-python-dependencies-in-pyspark.html

...but when I try the packing works fine but I recieve a file not found error (see screenshot). What can I do to fix this?

Thanks in advance.

Re: Use of venv pack

Hubert-Dudek — Tue, 30 Aug 2022 20:16:37 GMT

There are no stupid questions, only ones not asking them 🙂

What package are you need on the cluster? Maybe it can be installed using ready out-of.the-box options.

Re: Use of venv pack

c038644 — Wed, 31 Aug 2022 05:28:21 GMT

Thanks for your answer, I tried but it didn't make any differerence (see attached). I will look again at my environment settings as as the error is coming back with '\\' maybe I just need to delete a '\' somewhere?

Re: Use of venv pack

Debayan — Wed, 31 Aug 2022 04:54:24 GMT

Can you try using an absolute path instead of a relative path for the same?

For example: https://stackoverflow.com/questions/38661464/filenotfounderror-winerror-3