cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

python library in databricks

chari
Contributor

Hello community members,

I am seeking to understand where databricks keeps all the python libraries ? 

For a start, I tried two lines below:

import sys 

sys.path()

This list all the paths but I cant look inside them. How is DBFS different from these paths from sys.path() ?

Regards,

Veeru

1 REPLY 1

Wojciech_BUK
Valued Contributor III

Hello,
all your libraries are installed on Databricks Cluster Driver node on OS Disk.
DBFS is like mounted Cloude Storage account.

You have veriouse ways of working with libraries but databricks only load some of libraries that comes with Cluster image.

If you want custom librariesm, you can put them e.g. to DBFS and when you launch cluster, you can install those libraries on your cluster from DBFS (or use init script).

If you install libraries from inside of notebook, this installation will persist as long as your cluster won't be terminated. 
Next time cluster starts, you need to install those libraries again as it is brand new compute with brand new disk. 

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!