cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

How to use/access in a python notebook a scala library installed from JAR file?

blackcoffeeAR
Contributor

I'm using Azure Event Hubs Connector https://github.com/Azure/azure-event-hubs-spark to connect an Even Hub.

When I install this library from Maven , then everything works, I can access lib classes using JVM:

connection_string = "<connection_string>"
sc._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(connection_string)

However for some reasons I have to istall the same lib from previously downloaded JAR file. The file is downloaded from https://search.maven.org/artifact/com.microsoft.azure/azure-eventhubs-spark_2.12/2.3.22/jar. But then I cannot access the lib classes:

ERROR:root:Exception while sending command.
Traceback (most recent call last):
  File "/databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/clientserver.py", line 516, in send_command
    raise Py4JNetworkError("Answer from Java side is empty")
py4j.protocol.Py4JNetworkError: Answer from Java side is empty
 
During handling of the above exception, another exception occurred:
 
Traceback (most recent call last):
  File "/databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1038, in send_command
    response = connection.send_command(command)
  File "/databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/clientserver.py", line 539, in send_command
    raise Py4JNetworkError(
py4j.protocol.Py4JNetworkError: Error while sending or receiving
Py4JError: org.apache.spark.eventhubs.EventHubsUtils.encrypt does not exist in the JVM
---------------------------------------------------------------------------
Py4JError                                 Traceback (most recent call last)
<command-110542307469722> in <cell line: 17>()
     15 
     16 connectionString = ""
---> 17 sc._gateway.jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(connectionString)
 
/databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py in __getattr__(self, name)
   1545                     answer, self._gateway_client, self._fqn, name)
   1546         else:
-> 1547             raise Py4JError(
   1548                 "{0}.{1} does not exist in the JVM".format(self._fqn, name))
   1549 
 
Py4JError: org.apache.spark.eventhubs.EventHubsUtils.encrypt does not exist in the JVM

What I have tried is to import the lib but it did not help:

from py4j.java_gateway import java_import
java_import(sc._gateway.jvm,"org.apache.spark.eventhubs")

Thank for any hints.

2023-02-02 09_30_01-Window

17 REPLIES 17

-werners-
Esteemed Contributor III

Can you use this one which is built for python?

https://pypi.org/project/azure-eventhub/

Thanks, yes I could use the python library if I rework my application. But it is then some effort.

Anyway it is interesting why the same library installed from .jar file instead of MAven can not be used.

Anonymous
Not applicable

Hi @blackcoffee AR​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

Hi @Vidula Khanna​ ,

no, there is no solution.

BR

Kaniz
Community Manager
Community Manager

Hi @blackcoffee AR​,

  1. Did you try restarting your cluster?
  2. Ensure you have the correct versions of the libraries and that they are compatible with the version of Spark you are using.

Hi @Kaniz Fatma​ ,

  1. Restarting the cluster after installation does not help.
  2. Anything is compatible. JAR file downloaded from https://mvnrepository.com/artifact/com.microsoft.azure/azure-eventhubs-spark_2.12/2.3.22. PySpark Version :3.3.0. Cluster Runtime11.3.x-scala2.12

With only (1) one installed, it is not working. With (1) and (2) installed, everything is working.

2023-03-02 14_04_00-Window

Hi @blackcoffee AR​, Thank you for your question! To assist you better, please take a moment to review the answers and let me know which one best fits your needs.

Please select the best answer once your question is resolved.

Your feedback will help us ensure that we are providing the best possible service to you.

Thank you!

@Kaniz Fatma​ , there is no solution for the problem in the answers.

Hi @blackcoffee AR​ , I'm sorry you could not find a solution to your problem in the answers provided.

Our community strives to provide helpful and accurate information, but sometimes an immediate solution may not be available for every issue.

I suggest providing more information about your problem, such as specific error messages, error logs or details about the steps you have taken. This can help our community members better understand the issue and provide more targeted solutions.

Alternatively, you can consider contacting the support team for your product or service. They may be able to provide additional assistance or escalate the issue to the appropriate section for further investigation.

Thank you for your patience and understanding, and please let us know if there is anything else we can do to assist you.

Kaniz
Community Manager
Community Manager

Hi @blackcoffee AR​, please provide the library names you downloaded.

Hi @blackcoffee AR​, Thank you. We appreciate your patience and understand this is an urgent matter for you.

Our team is currently working to find a solution to your problem, and we apologize for any inconvenience this may have caused. We kindly ask for your patience and understanding as we work to resolve this issue as quickly as possible.

We will keep you updated on any progress or updates regarding your issue. If you have any additional information or updates that may help us find a solution, please do not hesitate to share them with us.

Thank you for your understanding and cooperation, and we will do our best to find a solution for you as soon as possible.

Code to reproduce:

sc._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt("1234567890")

Error message:

ERROR:root:Exception while sending command.
Traceback (most recent call last):
  File "/databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/clientserver.py", line 516, in send_command
    raise Py4JNetworkError("Answer from Java side is empty")
py4j.protocol.Py4JNetworkError: Answer from Java side is empty
 
During handling of the above exception, another exception occurred:
 
Traceback (most recent call last):
  File "/databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1038, in send_command
    response = connection.send_command(command)
  File "/databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/clientserver.py", line 539, in send_command
    raise Py4JNetworkError(
py4j.protocol.Py4JNetworkError: Error while sending or receiving
Py4JError: org.apache.spark.eventhubs.EventHubsUtils.encrypt does not exist in the JVM
---------------------------------------------------------------------------
Py4JError                                 Traceback (most recent call last)
<command-1516643832031051> in <cell line: 1>()
----> 1 sc._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(connectionString)
 
/databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py in __getattr__(self, name)
   1545                     answer, self._gateway_client, self._fqn, name)
   1546         else:
-> 1547             raise Py4JError(
   1548                 "{0}.{1} does not exist in the JVM".format(self._fqn, name))
   1549 
 
Py4JError: org.apache.spark.eventhubs.EventHubsUtils.encrypt does not exist in the JVM

Hi @blackcoffee AR​, Thank you for providing us with the code to reproduce the issue you are facing. Your effort is greatly appreciated; it has helped us better understand the problem.

We will now analyze the code you provided and work on finding a solution. We will update you on our progress and let you know once we have a resolution.

Please get in touch with us with any further information or questions about the issue.

Thank you again for your assistance.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.