Databricks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-30-2025 03:11 AM
Hi Everyone,
I have implemented Auto loader working as expected. i need to track the files which are loaded into stage table.
Here is the issue, the file tracking table need to create in snowflake from here i need to track the files.
How to connect databricks and snowflake, pls suggest.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-30-2025 05:12 AM - edited 04-30-2025 05:17 AM
You can connect Databricks to Snowflake using the Snowflake Spark connector. Once the connection is in place, you can use Auto Loader to capture file metadata and write it into a Snowflake table for tracking.
- Set up the Snowflake connection
sf_options = {
"sfURL": "<your_snowflake_account>.snowflakecomputing.com",
"sfWarehouse": "<warehouse>",
"sfDatabase": "<database>",
"sfSchema": "<schema>",
"sfUser": "<username>",
"sfPassword": "<password>"
}
- Write file tracking data to Snowflake
tracking_df.write.format("snowflake") \
.options(**sf_options) \
.option("dbtable", "<file_tracking_table>") \
.mode("append") \
.save()
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-30-2025 05:32 AM
Thanks for your reply, will try and let you know
