topic Trigger a Databricks Job When there is an insert to a Snowflake Table? in Data Engineering

Trigger a Databricks Job When there is an insert to a Snowflake Table?

abelian-grape — Wed, 12 Mar 2025 17:51:15 GMT

I need to automatically trigger a Databricks job whenever a new row is inserted to a Snowflake table. Additionally, I need the job to receive the exact details of the newly inserted row as parameters.

What are the best approaches to achieve this? I’m considering options like Snowflake Streams, AWS EventBridge, Lambda, or any other efficient method that integrates well with Databricks.

Has anyone implemented something similar?

Any guidance or best practices would be greatly appreciated!

Re: Trigger a Databricks Job When there is an insert to a Snowflake Table?

ashraf1395 — Thu, 13 Mar 2025 04:40:30 GMT

I think lamba function/ event bridge would be a good way - You can query your snowflake table there and create logic for any new row insert mabe CDC etc and then you send a job trigger using databricks API / databricks SDK
where you can pass your newly inserted rows as job parameters.
https://docs.databricks.com/api/workspace/jobs/runnow

I don't know the exact purpose of the job but if you want to be in databricks environment completely you can also try lakehouse federation for snowflake.
- this way you can bring your table as foreign catalog in databricks and
- then maybe create a materialized view on top of it with cdf enabled ,
- and set file trigger for the location where the mv table writes and then create your job whenever anything will be written in the mv table your job will trigger.
i don't know how exactly will it work but if you want to be in databricks ecosystem lakehouse federation with snowflake can be given a try or else first option is better.
https://docs.databricks.com/aws/en/query-federation/snowflake

Re: Trigger a Databricks Job When there is an insert to a Snowflake Table?

abelian-grape — Fri, 14 Mar 2025 13:30:46 GMT

Is there any reason no to simply use databricks workflow with a continous trigger to consume from a snowflake stream?

Re: Trigger a Databricks Job When there is an insert to a Snowflake Table?

abelian-grape — Sat, 15 Mar 2025 02:50:52 GMT

You cannot read a change data feed from a materialized view.

According to https://docs.databricks.com/aws/en/views/materialized

Re: Trigger a Databricks Job When there is an insert to a Snowflake Table?

ashraf1395 — Sat, 15 Mar 2025 03:48:28 GMT

Hi @abelian-grape , My bad - it will be either a streaming or managed table

Re: Trigger a Databricks Job When there is an insert to a Snowflake Table?

abelian-grape — Mon, 17 Mar 2025 16:33:07 GMT

what do you mean by "set file trigger for the location where the mv table writes and then create your job whenever anything will be written in the mv table your job will trigger." ?

Isn't the file trigger for external location such as s3? It will work for snowflake foreign catalog?