Streaming Live Table - What is actually computed?

tliuzillow
New Contributor

Can anyone please share in a DLT or structured streaming task, what group of rows are computed?

Specific scenarios:

1. when a streaming table A joining a delta table B. Is each of the minibatches in A joining the whole delta table? Does Spark compute the joining from each minibatch with the whole table B?

2. when a streaming table A joining another streaming table B.  Does Spark compute the joining from only the new minibatch in A with minibatch in B?  or the whole table A is joining the whole table B?

Thanks