topic Re: Databricks in Data Engineering

Databricks

RameshChejarla — Wed, 30 Apr 2025 10:11:52 GMT

Hi Everyone,

I have implemented Auto loader working as expected. i need to track the files which are loaded into stage table.

Here is the issue, the file tracking table need to create in snowflake from here i need to track the files.

How to connect databricks and snowflake, pls suggest.

Re: Databricks

SP_6721 — Wed, 30 Apr 2025 12:17:38 GMT

Hi @RameshChejarla

You can connect Databricks to Snowflake using the Snowflake Spark connector. Once the connection is in place, you can use Auto Loader to capture file metadata and write it into a Snowflake table for tracking.

- Set up the Snowflake connection

sf_options = {
"sfURL": "<your_snowflake_account>.snowflakecomputing.com",
"sfWarehouse": "<warehouse>",
"sfDatabase": "<database>",
"sfSchema": "<schema>",
"sfUser": "<username>",
"sfPassword": "<password>"
}

- Write file tracking data to Snowflake

tracking_df.write.format("snowflake") \
.options(**sf_options) \
.option("dbtable", "<file_tracking_table>") \
.mode("append") \
.save()

Re: Databricks

RameshChejarla — Wed, 30 Apr 2025 12:32:10 GMT

Thanks for your reply, will try and let you know