How to install python package on spark cluster

[kidexp]

Hi,

How can I install python packages on spark cluster? in local, I can use pip install.

I want to use some external packages which is not installed on was spark cluster.

Thanks for any suggestions.

[arsalan1 · Accepted solution]

@kidexp​ 

From the workspace dropdown, you can select New Library, and then select Python eggs or specify specific packages. Please see attached screenshots.

 

[kidexp]

Thanks very much @Arsalan Tavakoli-Shiraji​ 

[ChristianRodrig]

@Arsalan Tavakoli-Shiraji​  how do we attach it to a specific cluster programmatically (and not just all clusters by checking that box)

[dillon_bostwick]

You can use the Databricks Libraries API to programmatically attach libraries to specific clusters. For more information: https://docs.databricks.com/api/latest/libraries.html#install

[Anonymous]

Introduce Python bundle on flash group

Make a virtualenv only for your Flash hubs.

Each time you run a Flash work, run a new pip introduce of all your own in-house Python libraries. ...

Zoom up the site-bundles dir of the virtualenv. ...

Pass the single .compress document, containing your libraries and their conditions as a contention to - - py-records.

Custom Boxes With Logo | Custom Labels and Stickers