cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to install python package on spark cluster

kidexp
New Contributor II

Hi,

How can I install python packages on spark cluster? in local, I can use pip install.

I want to use some external packages which is not installed on was spark cluster.

Thanks for any suggestions.

1 ACCEPTED SOLUTION

Accepted Solutions

arsalan1
Contributor

@kidexp​ 

From the workspace dropdown, you can select New Library, and then select Python eggs or specify specific packages. Please see attached screenshots.

screen-shot-2015-04-14-at-30305-pm.png34-screen-shot-2015-04-14-at-30248-pm 

View solution in original post

5 REPLIES 5

arsalan1
Contributor

@kidexp​ 

From the workspace dropdown, you can select New Library, and then select Python eggs or specify specific packages. Please see attached screenshots.

screen-shot-2015-04-14-at-30305-pm.png34-screen-shot-2015-04-14-at-30248-pm 

kidexp
New Contributor II

Thanks very much @Arsalan Tavakoli-Shiraji​ 

@Arsalan Tavakoli-Shiraji​  how do we attach it to a specific cluster programmatically (and not just all clusters by checking that box)

You can use the Databricks Libraries API to programmatically attach libraries to specific clusters. For more information: https://docs.databricks.com/api/latest/libraries.html#install

Anonymous
Not applicable

Introduce Python bundle on flash group

Make a virtualenv only for your Flash hubs.

Each time you run a Flash work, run a new pip introduce of all your own in-house Python libraries. ...

Zoom up the site-bundles dir of the virtualenv. ...

Pass the single .compress document, containing your libraries and their conditions as a contention to - - py-records.

Custom Boxes With Logo | Custom Labels and Stickers

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!