Databricks Community

RameshChejarla · 3 weeks ago

Hi Everyone,

I have implemented Auto loader working as expected. i need to track the files which are loaded into stage table.

Here is the issue, the file tracking table need to create in snowflake from here i need to track the files.

How to connect databricks and snowflake, pls suggest.

SP_6721 · 3 weeks ago

You can connect Databricks to Snowflake using the Snowflake Spark connector. Once the connection is in place, you can use Auto Loader to capture file metadata and write it into a Snowflake table for tracking.

- Set up the Snowflake connection

sf_options = {
"sfURL": "<your_snowflake_account>.snowflakecomputing.com",
"sfWarehouse": "<warehouse>",
"sfDatabase": "<database>",
"sfSchema": "<schema>",
"sfUser": "<username>",
"sfPassword": "<password>"
}

- Write file tracking data to Snowflake

tracking_df.write.format("snowflake") \
.options(**sf_options) \
.option("dbtable", "<file_tracking_table>") \
.mode("append") \
.save()