Upgrading Ipython version without changing LTS version

[Hitesh_goswami]

I am using a specific Pydeeque function called ColumnProfilerRunner which is only supported with Spark 3.0.1, so I must use 7.3 LTS. Currently, I am trying to install "great_expectations" library on Python, which requires Ipython version==7.16.3, and it's failing because version 7.3LTS comes with Ipython version==7.12.0.

How can I upgrade Ipthon version on a 7.3LTS cluster?

[Anonymous]

@Hitesh Goswami​ : please check if the below helps!

To upgrade the Ipython version on a Databricks 7.3LTS cluster, you can follow these steps:

1. Create a new library installation command using the Databricks CLI by running the following command in your local terminal or command prompt:

databricks libraries install --cluster-id <cluster-id> --pypi-package ipython --pypi-package-version <desired-version>

Replace <cluster-id> with the ID of your Databricks cluster and <desired-version> with the desired version of Ipython that you want to install.

Once the library installation command is created, run it using the Databricks CLI by running the following command in your local terminal or command prompt:

databricks libraries install --cluster-id <cluster-id> --pypi-package ipython --pypi-package-version <desired-version> --whl <path-to-wheel-file>
 

Replace <cluster-id> with the ID of your Databricks cluster, <desired-version> with the desired version of Ipython that you want to install, and <path-to-wheel-file> with the local path to the wheel file for the desired version of Ipython.

Note that the wheel file for the desired version of Ipython can be found on the PyPI website.

Restart the cluster to apply the changes.