Hubert-Dudek
Databricks MVP

To avoid conflict with preinstalled version, py4j needs to be installed via %pip install py4j==0.10.9.

you can try this way to check where it is installed:

%sh
pip install py4j==0.10.9
pip show py4j


My blog: https://databrickster.medium.com/