I am using a specific Pydeeque function called ColumnProfilerRunner which is only supported with Spark 3.0.1, so I must use 7.3 LTS. Currently, I am trying to install "great_expectations" library on Python, which requires Ipython version==7.16.3, and it's failing because version 7.3LTS comes with Ipython version==7.12.0.

How can I upgrade Ipthon version on a 7.3LTS cluster?