cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

What is the difference between a trigger once stream and a normal one time write?

User16826992666
Valued Contributor

It seems to me like both of these would accomplish the same thing in the end. Do they use different mechanisms to accomplish it though? Are there any hidden costs to streaming to consider?

1 ACCEPTED SOLUTION

Accepted Solutions

Ryan_Chynoweth
Esteemed Contributor

The biggest reason to use the streaming API over the non-stream API would be to enable the checkpoint log to maintain a processing log. It is most common for people to use the trigger once when they want to only process the changes between executions (CDC processing).

As of recently delta does over CDC between versions of delta tables so this may limit the trigger once use cases but there could be other common use cases that could be out there.

There are no additional costs for streaming as everything is charged at the DBU level. If the batch process takes 30 seconds and a trigger once stream executes in 30 seconds then both will be charged the same.

View solution in original post

1 REPLY 1

Ryan_Chynoweth
Esteemed Contributor

The biggest reason to use the streaming API over the non-stream API would be to enable the checkpoint log to maintain a processing log. It is most common for people to use the trigger once when they want to only process the changes between executions (CDC processing).

As of recently delta does over CDC between versions of delta tables so this may limit the trigger once use cases but there could be other common use cases that could be out there.

There are no additional costs for streaming as everything is charged at the DBU level. If the batch process takes 30 seconds and a trigger once stream executes in 30 seconds then both will be charged the same.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group