topic Re: Getting Py4J "Could not find py4j jar" error when trying to use pypmml, solution doesn't work in Machine Learning

Getting Py4J "Could not find py4j jar" error when trying to use pypmml, solution doesn't work

mattsteinpreis — Tue, 03 May 2022 00:45:19 GMT

I'm trying to use pypmml in a DB notebook, but I'm getting the known `Error : Py4JError: Could not find py4j jar at` error. I've followed the solution here: https://kb.databricks.com/libraries/pypmml-fail-find-py4j-jar.html. However, this has not worked for me.

Details:

When I run `%pip install py4j==0.10.9` followed by `%sh find /databricks/ -name "py4j*jar"`, no results are found. However, if I install to the cluster via the Compute UI, then I do find the jar in the expected path.
I move the jar via:

dbutils.fs.cp('/databricks/python3/share/py4j/py4j0.10.9.jar', '/py4j/')

I create the init script, like so:

dbutils.fs.put("/<my-path>/install-py4j-jar.sh", """
 
#!/bin/bash
mkdir -p /share/py4j/ /current-release/
cp /dbfs/py4j/py4j0.10.9.jar /share/py4j/
cp /dbfs/py4j/py4j0.10.9.jar /current-release
""", True)

I attach init script, and restart.
I install pypmml and run something like:

from pypmml import Model
model = Model.load('/dbfs/<my-path>/<my-model>.pmml')

I've tried installing pypmml using %pip as well as in the cluster UI.

No matter what, I always get the same error: Py4JError: Could not find py4j jar at

I'm using DRV 10.4 LTS ML, though I've tried other versions to no avail.

Any ideas?

Re: Getting Py4J "Could not find py4j jar" error when trying to use pypmml, solution doesn't work

Hubert-Dudek — Tue, 03 May 2022 14:09:03 GMT

To avoid conflict with preinstalled version, py4j needs to be installed via %pip install py4j==0.10.9.

you can try this way to check where it is installed:

%sh
pip install py4j==0.10.9
pip show py4j

Re: Getting Py4J "Could not find py4j jar" error when trying to use pypmml, solution doesn't work

mattsteinpreis — Tue, 03 May 2022 16:02:49 GMT

This didn't fix the problem.

When I do this, pypmml does "see" this version, as when I later install pypmml, it skips the py4j requirement install and cites the py4j location from the show command. However, I still get the same error.

I don't know how to make pypmml know where to look to find the right py4j jar.

Also, even when I install py4j in this way, the databricks environment still seems to point to a different py4j install. If I run:

import py4j
print(py4j.__version__)
print(py4j.__file__)

I get a different version and path than what was specified/returned from the install commands.

Re: Getting Py4J "Could not find py4j jar" error when trying to use pypmml, solution doesn't work

jose_gonzalez — Fri, 29 Jul 2022 00:22:12 GMT

Hi @Matthew Steinpreis,

Just a friendly follow-up. Are you still looking for help? please let us know

Re: Getting Py4J "Could not find py4j jar" error when trying to use pypmml, solution doesn't work

pawelmitrus — Tue, 02 Aug 2022 15:47:37 GMT

I've been struggling myslef with it, but after installing pypmml for spark, I can use the other library, maybe it will work for you:

runtime 10.4 LTS ML
install pypmml-spark (https://github.com/autodeployai/pypmml-spark)
install pmml4s-spark (org.pmml4s:pmml4s-spark_2.12:0.9.16, pyspark is eventually using it, https://github.com/autodeployai/pmml4s-spark)

both pyspark & scala works