topic How to install (mssql) drivers to jobcompute? in Administration & Architecture

How to install (mssql) drivers to jobcompute?

PabloCSD — Thu, 24 Apr 2025 20:53:54 GMT

Hello, I'm having this issue with job-computes:

The snippet of the code is as follows:

84 if self.conf["persist_to_sql"]: 85 # persist to sql 86 df_parsed.write.format( 87 "com.microsoft.sqlserver.jdbc.spark" 88 ).option("url", self.get_sql_connection_string()).option( 89 "dbtable", self.conf["sql_tables"]["report"] 90 ).option( 91 "driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver" 92 ).mode( 93 "append" ---> 94 ).save() 95 self.logger.info( 96 f"Saved to {self.conf['sql_tables']['report']}" 97 )

How can I install the driver to this job-compute?

This is the driver I'm want to install: spark-mssql-connector_2.12-1.4.0-BETA.jar

I tried:

Upload to the workspace shared folder an init script that installs the driver (didn't work)
Indicate to the databricks.yml configuration to install the library at the cluster definition from maven

Thanks

Re: How to install (mssql) drivers to jobcompute?

PabloCSD — Mon, 28 Apr 2025 22:31:03 GMT

Up!

Re: How to install (mssql) drivers to jobcompute?

AIenthu — Tue, 29 Apr 2025 03:07:39 GMT

To install the spark-mssql-connector_2.12-1.4.0-BETA.jar driver for Azure Databricks job clusters, use one of these methods:

1. Cluster-Level Installation via Maven

Add these Maven coordinates to your cluster/library configuration:

text

com.microsoft.azure:spark-mssql-connector_2.12:1.4.0-BETA com.microsoft.sqlserver:mssql-jdbc:8.4.1.jre8

Steps:
1. Navigate to your Databricks cluster settings.
2. Under Libraries, select Install New → Maven.
3. Paste the coordinates above and install

2. Job-Specific Configuration

For job clusters, specify libraries directly in the job settings:

text

libraries: - maven: coordinates: "com.microsoft.azure:spark-mssql-connector_2.12:1.4.0-BETA" - maven: coordinates: "com.microsoft.sqlserver:mssql-jdbc:8.4.1.jre8"

This ensures dependencies are included when the job cluster starts

3. Init Script (Alternative)

If using an init script, ensure both the connector and JDBC driver are copied:

bash

#!/bin/bash cp /dbfs/FileStore/jars/spark-mssql-connector_2.12-1.4.0-BETA.jar /databricks/jars/ cp /dbfs/FileStore/jars/mssql-jdbc-8.4.1.jre8.jar /databricks/jars/

Upload the JARs to DBFS first

Key Considerations

Spark Version Compatibility: Confirm your Databricks runtime aligns with the connector’s Spark 3.4 requirement
JDBC Dependency: The mssql-jdbc driver is mandatory to resolve NoClassDefFoundError
Avoid Manual DBFS Uploads: Maven installation is preferred for dependency management

For Azure Data Factory pipelines, configure the libraries in the Databricks linked service’s Advanced settings under "Libraries"

Re: How to install (mssql) drivers to jobcompute?

NandiniN — Thu, 01 May 2025 06:52:28 GMT

For a job compute, you would have to go init script route.

Can you please highlight, the cause of the failure of library installation via init script?

Re: How to install (mssql) drivers to jobcompute?

PabloCSD — Mon, 05 May 2025 13:58:48 GMT

Hello @NandiniN ,

In the Shared folder I created a init_script for the JobCompute defined as its shown:

#!/bin/bash echo "=====> Installing MSSQL Spark Connector" cp /dbfs/Workspace/Shared/drivers/spark-mssql-connector_2.12-1.4.0-BETA.jar /databricks/jars/ echo "=====> Connector copied!"

In the logs there isn't any log and the workflow is still failing due to the missing driver, if you could give me any direction for trying to solve this would be much appreciated

Re: How to install (mssql) drivers to jobcompute?

PabloCSD — Tue, 06 May 2025 21:32:21 GMT

Up!

Re: How to install (mssql) drivers to jobcompute?

PabloCSD — Tue, 20 May 2025 14:53:15 GMT

@NandiniNplease check

Re: How to install (mssql) drivers to jobcompute?

PabloCSD — Tue, 27 May 2025 20:06:30 GMT

@NandiniNplease check