https://community.databricks.com/t5/data-engineering/pyspark-environment-and-python-packages-on-executors/m-p/15863#M10136 <P>databricks provides library installation in the form of PyPi packages, or wheel/egg.</P><P>If you install the packages like that on the cluster, they are automatically sent to all executors.</P> Wed, 29 Jun 2022 10:52:33 GMT -werners- 2022-06-29T10:52:33Z https://community.databricks.com/t5/data-engineering/pyspark-environment-and-python-packages-on-executors/m-p/15862#M10135 <P>At my company we use conda-pack to make certain packages available on the spark executors. Is there a better a better alternative to get away from creating a new environment and pack it every time I need a new python lib to be available for the executors?</P><P>​</P><P>Cheers <span class="lia-unicode-emoji" title=":slightly_smiling_face:">🙂</span></P><P>​</P> Wed, 29 Jun 2022 02:27:35 GMT https://community.databricks.com/t5/data-engineering/pyspark-environment-and-python-packages-on-executors/m-p/15862#M10135 rbarata 2022-06-29T02:27:35Z https://community.databricks.com/t5/data-engineering/pyspark-environment-and-python-packages-on-executors/m-p/15863#M10136 <P>databricks provides library installation in the form of PyPi packages, or wheel/egg.</P><P>If you install the packages like that on the cluster, they are automatically sent to all executors.</P> Wed, 29 Jun 2022 10:52:33 GMT https://community.databricks.com/t5/data-engineering/pyspark-environment-and-python-packages-on-executors/m-p/15863#M10136 -werners- 2022-06-29T10:52:33Z