topic Re: Databricks default python libraries list & version in Data Engineering

Databricks default python libraries list & version

kjoth — Wed, 13 Oct 2021 12:58:37 GMT

We are using data-bricks. How do we know the default libraries installed in the databricks & what versions are being installed.

I have ran pip list, but couldn't find the pyspark in the returned list.

Re: Databricks default python libraries list & version

Prabakar — Wed, 13 Oct 2021 13:05:39 GMT

Hi @karthick J , you can refer to the release notes for this.

https://docs.databricks.com/release-notes/runtime/releases.html

To know which library and what version of that library are installed on the cluster, you can check the respective DBR version in the release notes which will give your the list of libraries that will be installed.

Re: Databricks default python libraries list & version

kjoth — Wed, 13 Oct 2021 13:16:01 GMT

Thanks for the info. But in this page, the python library pyspark package is not available. Am I missing something. Thanks for helping out

Re: Databricks default python libraries list & version

Hubert-Dudek — Wed, 13 Oct 2021 13:17:44 GMT

relating to checking spark version it can be useful as well:(pyspark on databricks is not installed via pip but version will match)

Re: Databricks default python libraries list & version

jose_gonzalez — Wed, 13 Oct 2021 18:00:08 GMT

Hi @karthick J ,

If you would like to see all the libraries installed in your cluster and the version, then I will recommend to check the "Environment" tab. In there you will be able to find all the libraries installed in your cluster.

Please follow these steps to access the Environment tab:

Navigate and open you cluster cluster view
Select "Spark UI" tab
Select the "Environment" sub tab. It will be inside. (I have attached a screenshot)

Re: Databricks default python libraries list & version

Erik — Mon, 18 Oct 2021 09:23:14 GMT

Neither pyspark,nor any of the other python pre-installed python libraries show up for me when I look there.