cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

What is the difference between a trigger once stream and a normal one time write?

User16826992666
Valued Contributor

It seems to me like both of these would accomplish the same thing in the end. Do they use different mechanisms to accomplish it though? Are there any hidden costs to streaming to consider?

1 ACCEPTED SOLUTION

Accepted Solutions

Ryan_Chynoweth
Honored Contributor III

The biggest reason to use the streaming API over the non-stream API would be to enable the checkpoint log to maintain a processing log. It is most common for people to use the trigger once when they want to only process the changes between executions (CDC processing).

As of recently delta does over CDC between versions of delta tables so this may limit the trigger once use cases but there could be other common use cases that could be out there.

There are no additional costs for streaming as everything is charged at the DBU level. If the batch process takes 30 seconds and a trigger once stream executes in 30 seconds then both will be charged the same.

View solution in original post

1 REPLY 1

Ryan_Chynoweth
Honored Contributor III

The biggest reason to use the streaming API over the non-stream API would be to enable the checkpoint log to maintain a processing log. It is most common for people to use the trigger once when they want to only process the changes between executions (CDC processing).

As of recently delta does over CDC between versions of delta tables so this may limit the trigger once use cases but there could be other common use cases that could be out there.

There are no additional costs for streaming as everything is charged at the DBU level. If the batch process takes 30 seconds and a trigger once stream executes in 30 seconds then both will be charged the same.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.