topic Re: How to use/access in a python notebook a scala library installed from JAR file? in Data Engineering

How to use/access in a python notebook a scala library installed from JAR file?

blackcoffeeAR — Thu, 02 Feb 2023 09:02:53 GMT

I'm using Azure Event Hubs Connector https://github.com/Azure/azure-event-hubs-spark to connect an Even Hub.

When I install this library from Maven , then everything works, I can access lib classes using JVM:

connection_string = "<connection_string>"
sc._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(connection_string)

However for some reasons I have to istall the same lib from previously downloaded JAR file. The file is downloaded from https://search.maven.org/artifact/com.microsoft.azure/azure-eventhubs-spark_2.12/2.3.22/jar. But then I cannot access the lib classes:

ERROR:root:Exception while sending command.
Traceback (most recent call last):
  File "/databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/clientserver.py", line 516, in send_command
    raise Py4JNetworkError("Answer from Java side is empty")
py4j.protocol.Py4JNetworkError: Answer from Java side is empty
 
During handling of the above exception, another exception occurred:
 
Traceback (most recent call last):
  File "/databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1038, in send_command
    response = connection.send_command(command)
  File "/databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/clientserver.py", line 539, in send_command
    raise Py4JNetworkError(
py4j.protocol.Py4JNetworkError: Error while sending or receiving
Py4JError: org.apache.spark.eventhubs.EventHubsUtils.encrypt does not exist in the JVM
---------------------------------------------------------------------------
Py4JError                                 Traceback (most recent call last)
<command-110542307469722> in <cell line: 17>()
     15 
     16 connectionString = ""
---> 17 sc._gateway.jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(connectionString)
 
/databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py in __getattr__(self, name)
   1545                     answer, self._gateway_client, self._fqn, name)
   1546         else:
-> 1547             raise Py4JError(
   1548                 "{0}.{1} does not exist in the JVM".format(self._fqn, name))
   1549 
 
Py4JError: org.apache.spark.eventhubs.EventHubsUtils.encrypt does not exist in the JVM

What I have tried is to import the lib but it did not help:

from py4j.java_gateway import java_import
java_import(sc._gateway.jvm,"org.apache.spark.eventhubs")

Thank for any hints.

Re: How to use/access in a python notebook a scala library installed from JAR file?

-werners- — Thu, 02 Feb 2023 14:57:34 GMT

Can you use this one which is built for python?

https://pypi.org/project/azure-eventhub/

Re: How to use/access in a python notebook a scala library installed from JAR file?

blackcoffeeAR — Thu, 02 Feb 2023 16:22:42 GMT

Thanks, yes I could use the python library if I rework my application. But it is then some effort.

Anyway it is interesting why the same library installed from .jar file instead of MAven can not be used.

Re: How to use/access in a python notebook a scala library installed from JAR file?

Anonymous — Thu, 09 Feb 2023 06:05:38 GMT

Hi @blackcoffee AR

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: How to use/access in a python notebook a scala library installed from JAR file?

blackcoffeeAR — Thu, 09 Feb 2023 06:45:34 GMT

Hi @Vidula Khanna ,

no, there is no solution.

Re: How to use/access in a python notebook a scala library installed from JAR file?

blackcoffeeAR — Thu, 02 Mar 2023 13:06:25 GMT

Hi @Kaniz Fatma ,

Restarting the cluster after installation does not help.
Anything is compatible. JAR file downloaded from https://mvnrepository.com/artifact/com.microsoft.azure/azure-eventhubs-spark_2.12/2.3.22. PySpark Version :3.3.0. Cluster Runtime11.3.x-scala2.12

With only (1) one installed, it is not working. With (1) and (2) installed, everything is working.

Re: How to use/access in a python notebook a scala library installed from JAR file?

blackcoffeeAR — Thu, 02 Mar 2023 15:13:29 GMT

@Kaniz Fatma , there is no solution for the problem in the answers.

Re: How to use/access in a python notebook a scala library installed from JAR file?

blackcoffeeAR — Thu, 02 Mar 2023 15:26:43 GMT

Hi @Kaniz Fatma ,

here:

https://github.com/Azure/azure-event-hubs-spark

https://mvnrepository.com/artifact/com.microsoft.azure/azure-eventhubs-spark_2.12/2.3.22

Re: How to use/access in a python notebook a scala library installed from JAR file?

blackcoffeeAR — Thu, 02 Mar 2023 16:12:57 GMT

Code to reproduce:

sc._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt("1234567890")

Error message:

ERROR:root:Exception while sending command.
Traceback (most recent call last):
  File "/databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/clientserver.py", line 516, in send_command
    raise Py4JNetworkError("Answer from Java side is empty")
py4j.protocol.Py4JNetworkError: Answer from Java side is empty
 
During handling of the above exception, another exception occurred:
 
Traceback (most recent call last):
  File "/databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1038, in send_command
    response = connection.send_command(command)
  File "/databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/clientserver.py", line 539, in send_command
    raise Py4JNetworkError(
py4j.protocol.Py4JNetworkError: Error while sending or receiving
Py4JError: org.apache.spark.eventhubs.EventHubsUtils.encrypt does not exist in the JVM
---------------------------------------------------------------------------
Py4JError                                 Traceback (most recent call last)
<command-1516643832031051> in <cell line: 1>()
----> 1 sc._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(connectionString)
 
/databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py in __getattr__(self, name)
   1545                     answer, self._gateway_client, self._fqn, name)
   1546         else:
-> 1547             raise Py4JError(
   1548                 "{0}.{1} does not exist in the JVM".format(self._fqn, name))
   1549 
 
Py4JError: org.apache.spark.eventhubs.EventHubsUtils.encrypt does not exist in the JVM

Re: How to use/access in a python notebook a scala library installed from JAR file?

blackcoffeeAR — Thu, 02 Mar 2023 20:01:47 GMT

I know how to create the Event hub configuration dictionary. The configuration is not the problem. Here is the bigger code snippet:

connectionString = "<exmaple>" # the value does not matter
ehConf = {
    "eventhubs.connectionString" : sc._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(connectionString)
}

Main problem is here:

Py4JError: org.apache.spark.eventhubs.EventHubsUtils.encrypt does not exist in the JVM

Re: How to use/access in a python notebook a scala library installed from JAR file?

Anonymous — Sun, 09 Apr 2023 02:12:54 GMT

Hi @blackcoffee AR

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!