topic 10.4LTS has outdated snowflake spark connector, how to force latest snowflake spark connector in Data Engineering

10.4LTS has outdated snowflake spark connector, how to force latest snowflake spark connector

DSam05 — Tue, 17 May 2022 19:54:10 GMT

Hi,

I am trying to run my code from a scala fatjar on azure-databricks, which connects to snowflake for the data.

I usually run my jar on 9.1 LTS.

However when I run on 10.4 LTS the performace was 4x degraded and in the log it says

WARN SnowflakeConnectorUtils$: query pushdown is not supported because you are using Spark 3.2.1 with a connector designed to support Spark 3.1. Either use the version of spark supported by the connector or install a version of the connector that supports your version of spark.

I checked the driver logs and it seems 10.4 LTS has a pre-installed version of snowflake spark connector net.snowflake:spark-snowflake_2.12:2.9.0-spark_3.1 (also mentioned here)

Although I have put the latest snowflake spark connector(net.snowflake:spark-snowflake_2.13:2.10.0-spark_3.2) in my fatjar databricks picks the pre-installed one only.

Anyone has faced this issue ? knows a solution for this.

Thanks

Re: 10.4LTS has outdated snowflake spark connector, how to force latest snowflake spark connector

Atanu — Thu, 26 May 2022 09:42:02 GMT

is lower DBR is working fine @Soumyajit Datta ?

Re: 10.4LTS has outdated snowflake spark connector, how to force latest snowflake spark connector

slavat — Sun, 08 Jan 2023 11:38:15 GMT

I also encountered the similar problem. This is a snippet from my log file:

22/12/18 09:36:28 WARN SnowflakeConnectorUtils$: Query pushdown is not supported because you are using Spark 3.2.0 with a connector designed to support Spark 3.1. Either use the version of Spark supported by the connector or install a version of the connector that supports your version of Spark.
22/12/18 09:36:29 INFO SparkConnectorContext$: Spark Connector system config: {
  "spark_connector_version" : "2.9.0",
  "spark_version" : "3.2.0",
  "application_name" : "Databricks Shell",
  "scala_version" : "2.12.14",
  "java_version" : "11.0.12",
  "jdbc_version" : "3.13.3",
  "certified_jdbc_version" : "3.13.3",
  "os_name" : "Linux",
......
}

Shouldn't the versionioning of Snowflake connector be transparent for me (I use the one supplied by Databricks)?