topic Re: Spark Sql Connector : in Data Engineering

Spark Sql Connector :

Rahul_Samant — Wed, 03 Aug 2022 09:03:06 GMT

i am trying to read data from azure sql database from databricks. azure sql database is created with private link endpoint.Using DBR 10.4 LTS Cluster and expectation is the connector is pre installed as per documentation.

using the below code to fetch but getting java.lang.ClassNotFoundException:

df = spark.read \

.format("com.microsoft.sqlserver.jdbc.spark") \

.option("url", "xyz-server.database.windows.net") \

.option("dbtable", "test") \

.option("databaseName","test") \

.option("user", 'test') \

.option("password", 'abc') \

.load()

Any help is appreciated

Re: Spark Sql Connector :

artsheiko — Wed, 03 Aug 2022 18:11:03 GMT

It seems that .option("databaseName", "test") is redundant here as you need to include the db name in the url.

Please verify that you use a connector compatible to your cluster's Spark version : Apache Spark connector: SQL Server & Azure SQL

Re: Spark Sql Connector :

Rahul_Samant — Thu, 04 Aug 2022 11:33:13 GMT

Hi @Artem Sheiko ,

i read its preinstalled in DBR Runtime and we don't have to worry about it ?

btw as per the link you shared . connector is only available for 3.1.x an DBR 10.4 LTS is 3.2.X. so that's why its not working ?

ConnectorMaven CoordinateSpark 2.4.x compatible connector

com.microsoft.azure:spark-mssql-connector:1.0.2

Spark 3.0.x compatible connector

com.microsoft.azure:spark-mssql-connector_2.12:1.1.0

Spark 3.1.x compatible connector

com.microsoft.azure:spark-mssql-connector_2.12:1.2.0

Re: Spark Sql Connector :

artsheiko — Thu, 04 Aug 2022 13:52:16 GMT

Hi,

My point was just to verify that you use a version compatible to DBR.

No, it's not preinstalled, you need to import it using maven coordinate. You can see here how proceed to install a it.

The list of installed libraries coul be find in a DBR 10.4 LTS release notes.

Re: Spark Sql Connector :

Rahul_Samant — Thu, 04 Aug 2022 13:58:15 GMT

i downgraded to 9.1 LTS and installed below . working now

Spark 3.1.x compatible connector

com.microsoft.azure:spark-mssql-connector_2.12:1.2.0