Streaming Live Table - What is actually computed?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-09-2024 09:30 PM
Can anyone please share in a DLT or structured streaming task, what group of rows are computed?
Specific scenarios:
1. when a streaming table A joining a delta table B. Is each of the minibatches in A joining the whole delta table? Does Spark compute the joining from each minibatch with the whole table B?
2. when a streaming table A joining another streaming table B. Does Spark compute the joining from only the new minibatch in A with minibatch in B? or the whole table A is joining the whole table B?
Thanks