topic Re: Need guidance on connecting to Azure Databricks using JDBC Protocol in Data Engineering

Need guidance on connecting to Azure Databricks using JDBC Protocol

subhankar — Wed, 08 Jan 2025 07:48:40 GMT

Step 1: Download and Reference the JDBC Driver

Download the Databricks JDBC Driver:

Visit the Databricks JDBC Driver download page.

Download the appropriate version for your operating system.

Extract the DatabricksJDBC42.jar file from the downloaded zip file.

Step 2: Gather Configuration Settings

Gather Configuration Settings:

Server Hostname: The hostname of your Azure Databricks cluster or SQL warehouse.

Port: The port number (default is 443).

HTTP Path: The HTTP path to your Databricks SQL endpoint.

Authentication Token: A personal access token or Microsoft Entra ID (formerly Azure Active Directory) token.

Step 3: Create a Personal Access Token

Create a Personal Access Token:

In your Azure Databricks workspace, click your username in the top bar and select Settings.

Click Developer.

Next to Access tokens, click Manage.

Click Generate new token.

Optionally, enter a comment and set the token's lifetime.

Click Generate and copy the token to a secure location.

Step 4: Configure the JDBC Connection

Configure the JDBC Connection:

Use the following JDBC URL format:

jdbc:spark://<Server Hostname>:<Port>/;transportMode=http;ssl=1;httpPath=<HTTP Path>;AuthMech=3;UID=token;PWD=<Personal Access Token>

Replace <Server Hostname>, <Port>, <HTTP Path>, and <Personal Access Token> with your specific values.

Step 5: Connect Using JDBC

Connect Using JDBC:

Depending on your programming language or tool, use the JDBC URL to establish a connection.

%python

import jaydebeapi

jdbc_url = "jdbc:spark://<Server Hostname>:<Port>/;transportMode=http;ssl=1;httpPath=<HTTP Path>;AuthMech=3;UID=token;PWD=<Personal Access Token>"

driver_class = "com.simba.spark.jdbc.Driver"

jar_file = "/path/to/DatabricksJDBC42.jar"

conn = jaydebeapi.connect(driver_class, jdbc_url, jars=jar_file)

cursor = conn.cursor()

cursor.execute("SELECT * FROM my_table")

result = cursor.fetchall()

cursor.close()

conn.close()

I have followed the above steps to connect to Azure Databricks using JDBC protocol. I am using Python to connect but getting below error:

raise JVMNotFoundException("No JVM shared library file ({0}) "
jpype._jvmfinder.JVMNotFoundException: No JVM shared library file (jvm.dll) found. Try setting up the JAVA_HOME environment variable properly.

Please help me with this and also let me know if there is another way without having JVM. Thanks

Re: Need guidance on connecting to Azure Databricks using JDBC Protocol

Ayushi_Suthar — Wed, 08 Jan 2025 08:12:32 GMT

Hi @subhankar , Good Day!

Looking at the error you are getting here shows that it tries to find some kind of JVM file and probably refers to the JAVA_HOME variable to achieve it. It looks as if it is not set correctly in your Environment Variables.

To resolve this issue, you should ensure that Java is correctly installed and configured on your system. Specifically, you should check that the JAVA_HOME environment variable is set to the path of your Java installation directory.

I am also sharing a few public documents which might help you to resolve this issue : https://stackoverflow.com/questions/75656048/this-problem-appeared-with-the-jpyp-library-and-it-gives-me-the-suggestion-to-a

https://github.com/iiasa/message_ix/issues/35

Moreover, you can refer to our below document to connect with Databricks using JDBC Drivers: https://docs.databricks.com/en/integrations/jdbc-odbc-bi.html#:~:text=Databricks%20JDBC%20Driver%20Guide

Please let me know if this helps and leave a like if this information is useful, followups are appreciated.
Kudos
Ayushi

Re: Need guidance on connecting to Azure Databricks using JDBC Protocol

subhankar — Wed, 15 Jan 2025 06:56:01 GMT

Hi @Ayushi_Suthar

I have set up the JAVA_HOME. Also, I manually downloaded the JDBC .jar file, and added the downloaded .jar file to the Java classpath. But still , getting the error "An error occurred: Class com.simba.spark.jdbc.Driver is not found". Could you please confirm, If I need to install Simba ODBC driver as well? If yes, then do i need to do any further configuration setup as well? I already have 32bit Simba ODBC driver installed, but no 64bit. I am trying to connect from Python.