topic Slow read using Snowflake connector in Databricks. in Data Engineering

Slow read using Snowflake connector in Databricks.

peta — Mon, 23 Oct 2023 14:44:05 GMT

Hi,

I am trying to read table from Snowflake with Databricks native Snowflake jdbc connector. It is going well for small amount of data (100 rows), but if I am adding more (even just 1000 rows) the query does not finish. I was checking if the query finishes in Snowflake part and is takes just few miliseconds. Could you please advice where could be the problem?

Re: Slow read using Snowflake connector in Databricks.

peta — Mon, 30 Oct 2023 13:35:01 GMT

Hi @Retired_mod ,

Thank you for your reply.

1. Network should not be a problem, the communication is between 2 Azure subscriptions, but I am checking it with our Operation team just to be sure.

2.We are using the smallest Snowflake warehouse (XS) and it seems working ok, all the queries I am sending there through JDBC ends with Success on Snowlake side but in Databricks we do not get the data. Futhermore I am running there really small queries, do you think that for the communication would be bigger warehouse better?

3. We are now using snowflake-jdbc 3.13.29 which should be the latest one. Which connection parameters do you thing can be tuned for better performace?

4. How can I use parquet files instead of csv through JDBC?

Thanks.

Petra

Re: Slow read using Snowflake connector in Databricks.

sri123 — Thu, 15 Feb 2024 16:52:18 GMT

Hi @peta

Have your issue got resolved, I'm facing the similar issue. If your issue got resolved can you please post the steps that you followed to resolve your issue?

Thanks & Regards

Sri

Re: Slow read using Snowflake connector in Databricks.

liquido — Fri, 27 Sep 2024 07:08:24 GMT

Hi,

Have your issue got resolved, I'm facing the similar issue. If your issue got resolved can you please post the steps that you followed to resolve your issue?

Thanks