Hello, I'm having this issue with job-computes:
The snippet of the code is as follows:
84 if self.conf["persist_to_sql"]:
85 # persist to sql
86 df_parsed.write.format(
87 "com.microsoft.sqlserver.jdbc.spark"
88 ).option("url", self.get_sql_connection_string()).option(
89 "dbtable", self.conf["sql_tables"]["report"]
90 ).option(
91 "driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver"
92 ).mode(
93 "append"
---> 94 ).save()
95 self.logger.info(
96 f"Saved to {self.conf['sql_tables']['report']}"
97 )How can I install the driver to this job-compute?
This is the driver I'm want to install: spark-mssql-connector_2.12-1.4.0-BETA.jar
I tried:
- Upload to the workspace shared folder an init script that installs the driver (didn't work)
- Indicate to the databricks.yml configuration to install the library at the cluster definition from maven
Thanks
Hello @NandiniN ,
In the Shared folder I created a init_script for the JobCompute defined as its shown:
#!/bin/bash
echo "=====> Installing MSSQL Spark Connector"
cp /dbfs/Workspace/Shared/drivers/spark-mssql-connector_2.12-1.4.0-BETA.jar /databricks/jars/
echo "=====> Connector copied!"
In the logs there isn't any log and the workflow is still failing due to the missing driver, if you could give me any direction for trying to solve this would be much appreciated
