Pyspark environment and python packages on executors

rbarata
New Contributor II

‎06-28-2022 07:27 PM

At my company we use conda-pack to make certain packages available on the spark executors. Is there a better a better alternative to get away from creating a new environment and pack it every time I need a new python lib to be available for the executors?

​

Cheers 🙂

​

Labels:

-werners-
Esteemed Contributor III

‎06-29-2022 03:52 AM

databricks provides library installation in the form of PyPi packages, or wheel/egg.

If you install the packages like that on the cluster, they are automatically sent to all executors.

View solution in original post