cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Upgrading Ipython version without changing LTS version

Hitesh_goswami
New Contributor

I am using a specific Pydeeque function called ColumnProfilerRunner which is only supported with Spark 3.0.1, so I must use 7.3 LTS. Currently, I am trying to install "great_expectations" library on Python, which requires Ipython version==7.16.3, and it's failing because version 7.3LTS comes with Ipython version==7.12.0.

How can I upgrade Ipthon version on a 7.3LTS cluster?

1 REPLY 1

Anonymous
Not applicable

@Hitesh Goswami​ : please check if the below helps!

To upgrade the Ipython version on a Databricks 7.3LTS cluster, you can follow these steps:

  1. Create a new library installation command using the Databricks CLI by running the following command in your local terminal or command prompt:
databricks libraries install --cluster-id <cluster-id> --pypi-package ipython --pypi-package-version <desired-version>

Replace <cluster-id> with the ID of your Databricks cluster and <desired-version> with the desired version of Ipython that you want to install.

Once the library installation command is created, run it using the Databricks CLI by running the following command in your local terminal or command prompt:

databricks libraries install --cluster-id <cluster-id> --pypi-package ipython --pypi-package-version <desired-version> --whl <path-to-wheel-file>
 

Replace <cluster-id> with the ID of your Databricks cluster, <desired-version> with the desired version of Ipython that you want to install, and <path-to-wheel-file> with the local path to the wheel file for the desired version of Ipython.

Note that the wheel file for the desired version of Ipython can be found on the PyPI website.

Restart the cluster to apply the changes.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group