Writing to Snowflake from Databricks - sqlalchemy replacement?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-21-2024 10:31 AM
I am trying to migrate some complex python load processes into databricks. Our load processes currently use pandas and we're hoping to refactor into Spark soon. For now, I need to figure out how to alter our functions that get sqlalchemy connection engines so I can bring our libraries that use sqlalchemy over to databricks. I see that there is a databricks sqlalchemy library, but there also seems to be a fairly strong option for connecting to Snowflake using a spark session and a JDBC(I think?) connector. Do these spark JDBC sessions work in a similar way to sqlalchemy connection sessions?
Has anyone ever done this before? What is the well-traveled path on this kind of migration?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-21-2024 11:10 AM
@brian999 - spark-snowflake connector is inbuilt into the DBR. Please refer to the below article for examples.
Please let us know if this helps
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-21-2024 11:16 AM
Thank you for your response. I am familiar with this documentation, which is rather sparse and the db write part is only in scala from what I can tell. I need to know if, when I start that spark session the session with snowflake stays open, because we create temp tables that rely on being used within the same connection session. Also, I'd like to know see any example code of using the databricks sqlalchemy library to connect to snowflake.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-26-2024 11:33 AM
@brian999 - Below is an example of spark snowflake connector using python.
https://docs.databricks.com/_extras/notebooks/source/snowflake-python.html