Writing to Snowflake from Databricks - sqlalchemy replacement?

[brian999]

I am trying to migrate some complex python load processes into databricks. Our load processes currently use pandas and we're hoping to refactor into Spark soon. For now, I need to figure out how to alter our functions that get sqlalchemy connection engines so I can bring our libraries that use sqlalchemy over to databricks. I see that there is a databricks sqlalchemy library, but there also seems to be a fairly strong option for connecting to Snowflake using a spark session and a JDBC(I think?) connector. Do these spark JDBC sessions work in a similar way to sqlalchemy connection sessions?  

Has anyone ever done this before? What is the well-traveled path on this kind of migration?

[shan_chandra]

@brian999  -  spark-snowflake connector is inbuilt into the DBR. Please refer to the below article for examples. 

https://docs.databricks.com/en/connect/external-systems/snowflake.html#read-and-write-data-from-snow...

Please let us know if this helps

[brian999]

Thank you for your response. I am familiar with this documentation, which is rather sparse and the db write part is only in scala from what I can tell. I need to know if, when I start that spark session the session with snowflake stays open, because we create temp tables that rely on being used within the same connection session.  Also, I'd like to know see any example code of using the databricks sqlalchemy library to connect to snowflake.

[shan_chandra]

@brian999  - Below is an example of spark snowflake connector using python. 

https://docs.databricks.com/_extras/notebooks/source/snowflake-python.html