rushi29
New Contributor III

‎08-07-2024 09:17 AM

Thanks @Retired_mod for your response. Since, I also need to call stored procedures in the Azure SQL databases from Azure Databricks, I don't think the DataFrames solution would work. When using py4j, how would I create a connection object in Azure Databricks? I tried with various sample code online but none of them worked. I was trying something similar as below. Am I missing anything?

Thanks,

Rushi

from pyspark.sql import SparkSession
from py4j.java_gateway import java_import, JavaGateway, GatewayParameters

spark = SparkSession.builder.appName("AzureSQLStoredProcedure").getOrCreate()
gateway = JavaGateway()
print (gateway.jvm.DriverManager)
jdbc_url = "jdbc:sqlserver://xxx.database.windows.net:1433;databaseName=xxx;authentication=ActiveDirectoryMSI"
connection = gateway.jvm.DriverManager.getConnection(jdbc_url)
print (connection)
 
This code gives an error
Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:25333)
File /databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py:982, in GatewayClient._get_connection(self) 981 try: --> 982 connection = self.deque.pop() 983 except IndexError:
File /databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py:1177, in GatewayConnection.start(self) 1174 msg = "An error occurred while trying to connect to the Java "\ 1175 "server ({0}:{1})".format(self.address, self.port) 1176 logger.exception(msg) -> 1177 raise Py4JNetworkError(msg, e)

 

0 Kudos