python library in databricks

chari — Thu, 21 Dec 2023 15:50:48 GMT

Hello community members,

I am seeking to understand where databricks keeps all the python libraries ?

For a start, I tried two lines below:

import sys

sys.path()

This list all the paths but I cant look inside them. How is DBFS different from these paths from sys.path() ?

Regards,

Veeru

Re: python library in databricks

Wojciech_BUK — Fri, 22 Dec 2023 00:47:54 GMT

Hello,
all your libraries are installed on Databricks Cluster Driver node on OS Disk.
DBFS is like mounted Cloude Storage account.

You have veriouse ways of working with libraries but databricks only load some of libraries that comes with Cluster image.

If you want custom librariesm, you can put them e.g. to DBFS and when you launch cluster, you can install those libraries on your cluster from DBFS (or use init script).

If you install libraries from inside of notebook, this installation will persist as long as your cluster won't be terminated.
Next time cluster starts, you need to install those libraries again as it is brand new compute with brand new disk.

topic Re: python library in databricks in Get Started Discussions

python library in databricks

Re: python library in databricks