Databricks default python libraries list & version

kjoth · ‎10-13-2021

We are using data-bricks. How do we know the default libraries installed in the databricks & what versions are being installed.

I have ran pip list, but couldn't find the pyspark in the returned list.

Prabakar · ‎10-13-2021

Hi @karthick J , you can refer to the release notes for this.

https://docs.databricks.com/release-notes/runtime/releases.html

To know which library and what version of that library are installed on the cluster, you can check the respective DBR version in the release notes which will give your the list of libraries that will be installed.

View solution in original post

kjoth · ‎10-13-2021

Thanks for the info. But in this page, the python library pyspark package is not available. Am I missing something. Thanks for helping out

Hubert-Dudek · ‎10-13-2021

relating to checking spark version it can be useful as well:(pyspark on databricks is not installed via pip but version will match)

My blog: https://databrickster.medium.com/

jose_gonzalez · ‎10-13-2021

Hi @karthick J ,

If you would like to see all the libraries installed in your cluster and the version, then I will recommend to check the "Environment" tab. In there you will be able to find all the libraries installed in your cluster.

Please follow these steps to access the Environment tab:

Navigate and open you cluster cluster view
Select "Spark UI" tab
Select the "Environment" sub tab. It will be inside. (I have attached a screenshot)

Erik · ‎10-18-2021

Neither pyspark,nor any of the other python pre-installed python libraries show up for me when I look there.