- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-16-2021 08:34 PM
It seems to me like both of these would accomplish the same thing in the end. Do they use different mechanisms to accomplish it though? Are there any hidden costs to streaming to consider?
- Labels:
-
Structured streaming
-
Trigger
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-17-2021 05:03 PM
The biggest reason to use the streaming API over the non-stream API would be to enable the checkpoint log to maintain a processing log. It is most common for people to use the trigger once when they want to only process the changes between executions (CDC processing).
As of recently delta does over CDC between versions of delta tables so this may limit the trigger once use cases but there could be other common use cases that could be out there.
There are no additional costs for streaming as everything is charged at the DBU level. If the batch process takes 30 seconds and a trigger once stream executes in 30 seconds then both will be charged the same.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-17-2021 05:03 PM
The biggest reason to use the streaming API over the non-stream API would be to enable the checkpoint log to maintain a processing log. It is most common for people to use the trigger once when they want to only process the changes between executions (CDC processing).
As of recently delta does over CDC between versions of delta tables so this may limit the trigger once use cases but there could be other common use cases that could be out there.
There are no additional costs for streaming as everything is charged at the DBU level. If the batch process takes 30 seconds and a trigger once stream executes in 30 seconds then both will be charged the same.

