Delta Live Tables - ForeachBatch

[IsmaelHenzel1]

I am wondering how to create complex streaming queries using Delta Live Tables (DLT). I can't find a way to use foreachBatch with it, and this is causing me some difficulty. I need to create a window using a lag without a time range, which is not possible in structured streaming but is fully achievable using foreachBatch in structured streaming.

Is there a way to overcome this limitation?

Additionally, if I create a materialized view in DLT to achieve the same functionality as foreachBatch in Spark Structured Streaming, will it process only the new available batches automatically based on some change data feed, or will it recompute everything again? If it processes the batches incrementally, are materialized views in some way equivalent to the Spark foreachBatch?

[Brahmareddy · Accepted solution]

Hi @IsmaelHenzel1,

How are you doing today?

As per understanding, Consider using Delta Live Tables (DLT) materialized views to handle complex streaming logic as DLT doesn’t currently support foreachBatch. For windowing with lag, DLT materialized views can help, but without a direct time range in streaming, you may need to adjust your approach. If you create a materialized view, it will process data incrementally, similar to foreachBatch, using the underlying change data feed (CDF). This means it should only compute on new data rather than recomputing the entire dataset, giving you incremental processing similar to the behavior in foreachBatch.

Please give a try and let me know for any queries.

Regards,

Brahma