https://community.databricks.com/t5/data-engineering/writing-to-snowflake-from-databricks-sqlalchemy-replacement/m-p/64320#M32536 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/102604">@brian999</a>&nbsp; -&nbsp; spark-snowflake connector is inbuilt into the DBR. Please refer to the below article for examples.&nbsp;</P> <P><A href="https://docs.databricks.com/en/connect/external-systems/snowflake.html#read-and-write-data-from-snowflake" target="_blank">https://docs.databricks.com/en/connect/external-systems/snowflake.html#read-and-write-data-from-snowflake</A></P> <P>Please let us know if this helps</P> Thu, 21 Mar 2024 18:10:06 GMT shan_chandra 2024-03-21T18:10:06Z https://community.databricks.com/t5/data-engineering/writing-to-snowflake-from-databricks-sqlalchemy-replacement/m-p/64315#M32531 <P>I am trying to migrate some complex python load processes into databricks. Our load processes currently use pandas and we're hoping to refactor into Spark soon. For now, I need to figure out how to alter our functions that get sqlalchemy connection engines so I can bring our libraries that use sqlalchemy over to databricks. I see that there is a databricks sqlalchemy library, but there also seems to be a fairly strong option for connecting to Snowflake using a spark session and a JDBC(I think?) connector. Do these spark JDBC sessions work in a similar way to sqlalchemy connection sessions?&nbsp;&nbsp;</P><P>&nbsp;</P><P>Has anyone ever done this before? What is the well-traveled path on this kind of migration?</P> Thu, 21 Mar 2024 17:31:01 GMT https://community.databricks.com/t5/data-engineering/writing-to-snowflake-from-databricks-sqlalchemy-replacement/m-p/64315#M32531 brian999 2024-03-21T17:31:01Z https://community.databricks.com/t5/data-engineering/writing-to-snowflake-from-databricks-sqlalchemy-replacement/m-p/64320#M32536 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/102604">@brian999</a>&nbsp; -&nbsp; spark-snowflake connector is inbuilt into the DBR. Please refer to the below article for examples.&nbsp;</P> <P><A href="https://docs.databricks.com/en/connect/external-systems/snowflake.html#read-and-write-data-from-snowflake" target="_blank">https://docs.databricks.com/en/connect/external-systems/snowflake.html#read-and-write-data-from-snowflake</A></P> <P>Please let us know if this helps</P> Thu, 21 Mar 2024 18:10:06 GMT https://community.databricks.com/t5/data-engineering/writing-to-snowflake-from-databricks-sqlalchemy-replacement/m-p/64320#M32536 shan_chandra 2024-03-21T18:10:06Z https://community.databricks.com/t5/data-engineering/writing-to-snowflake-from-databricks-sqlalchemy-replacement/m-p/64324#M32538 <P>Thank you for your response. I am familiar with this documentation, which is rather sparse and the db write part is only in scala from what I can tell. I need to know if, when I start that spark session the session with snowflake stays open, because we create temp tables that rely on being used within the same connection session.&nbsp; Also, I'd like to know see any example code of using the databricks sqlalchemy library to connect to snowflake.</P> Thu, 21 Mar 2024 18:16:50 GMT https://community.databricks.com/t5/data-engineering/writing-to-snowflake-from-databricks-sqlalchemy-replacement/m-p/64324#M32538 brian999 2024-03-21T18:16:50Z https://community.databricks.com/t5/data-engineering/writing-to-snowflake-from-databricks-sqlalchemy-replacement/m-p/64698#M32634 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/102604">@brian999</a>&nbsp; - Below is an example of spark snowflake connector using python.&nbsp;</P> <P><A href="https://docs.databricks.com/_extras/notebooks/source/snowflake-python.html" target="_blank">https://docs.databricks.com/_extras/notebooks/source/snowflake-python.html</A></P> <P>&nbsp;</P> Tue, 26 Mar 2024 18:33:26 GMT https://community.databricks.com/t5/data-engineering/writing-to-snowflake-from-databricks-sqlalchemy-replacement/m-p/64698#M32634 shan_chandra 2024-03-26T18:33:26Z