Snowflake connector

Phani1 — Tue, 16 Jan 2024 05:13:46 GMT

Hi Team, Databricks recommends storing data in a cloud storage location, but if we directly connect to Snowflake using the Snowflake connector, will we face any performance issues?

Could you please suggest the best way to read a large volume of data from Snowflake to Databricks?

Re: Snowflake connector

Wojciech_BUK — Tue, 16 Jan 2024 09:31:54 GMT

If you read data from Snowflake in Spark using e.g. spark.read.jdbc it will be slow. This is because the data is loaded in a single step, and is therefore loaded by a single executor.

You need to somehow distributing the query among the spark executors, and assign each executor to read a subset of the result, eg. by adding WHERE conditions or limit-offset clauses, and distribute them among the executors.

JDBC method has also option to supply following information partitionColumn, lowerBound, upperBound, and numPartitions , then you can paralelize it.

Another way is to sync data into DeltaLake and run your query against delta table.

-------
UPDATE: there is also one more way but would require redesign on Snoflake end -> to create table in Snowflake as External Iceberg table and connect your Databricks job to Iceberg but that might be overkill.

Re: Snowflake connector

Phani1 — Tue, 16 Jan 2024 14:46:23 GMT

Thanks !!

topic Re: Snowflake connector in Data Engineering

Snowflake connector

Re: Snowflake connector

Re: Snowflake connector