Spark Sql Connector :

[Rahul_Samant]

i am trying to read data from azure sql database from databricks. azure sql database is created with private link endpoint.Using DBR 10.4 LTS Cluster and expectation is the connector is pre installed as per documentation.

using the below code to fetch but getting java.lang.ClassNotFoundException:

df = spark.read \

    .format("com.microsoft.sqlserver.jdbc.spark") \

    .option("url", "xyz-server.database.windows.net") \

    .option("dbtable", "test") \

    .option("databaseName","test") \

    .option("user", 'test') \

    .option("password", 'abc') \

    .load()

Any help is appreciated

[artsheiko]

It seems that .option("databaseName", "test") is redundant here as you need to include the db name in the url.

Please verify that you use a connector compatible to your cluster's Spark version : Apache Spark connector: SQL Server & Azure SQL

[Rahul_Samant]

Hi @Artem Sheiko​ ,

i read its preinstalled in DBR Runtime and we don't have to worry about it ?

btw as per the link you shared . connector is only available for 3.1.x an DBR 10.4 LTS is 3.2.X. so that's why its not working ?

ConnectorMaven CoordinateSpark 2.4.x compatible connector

com.microsoft.azure:spark-mssql-connector:1.0.2

Spark 3.0.x compatible connector

com.microsoft.azure:spark-mssql-connector_2.12:1.1.0

Spark 3.1.x compatible connector

com.microsoft.azure:spark-mssql-connector_2.12:1.2.0

[artsheiko · Accepted solution]

Hi,

My point was just to verify that you use a version compatible to DBR.

No, it's not preinstalled, you need to import it using maven coordinate. You can see here how proceed to install a it.

The list of installed libraries coul be find in a DBR 10.4 LTS release notes.

[Rahul_Samant]

i downgraded to 9.1 LTS and installed below . working now

Spark 3.1.x compatible connector

com.microsoft.azure:spark-mssql-connector_2.12:1.2.0