- 1864 Views
- 3 replies
- 1 kudos
10.4LTS has outdated snowflake spark connector, how to force latest snowflake spark connector
Hi,I am trying to run my code from a scala fatjar on azure-databricks, which connects to snowflake for the data.I usually run my jar on 9.1 LTS.However when I run on 10.4 LTS the performace was 4x degraded and in the log it says WARN SnowflakeConnect...
- 1864 Views
- 3 replies
- 1 kudos
- 1 kudos
I also encountered the similar problem. This is a snippet from my log file:22/12/18 09:36:28 WARN SnowflakeConnectorUtils$: Query pushdown is not supported because you are using Spark 3.2.0 with a connector designed to support Spark 3.1. Either use t...
- 1 kudos
- 1093 Views
- 1 replies
- 0 kudos
Does spark utilise a temporary stage when writing to snowflake? How does that work?
Folks , when I want to push data to snowflake I need to use a stage for files before copying data over. However, when I utilise the net.snowflake.spark.snowflake.Utils library and do a spark.write as in...spark.read.format("csv") .option("header", ...
- 1093 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes it uses a temporary stage. should have just looked in snowflake history
- 0 kudos