cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Delta Live Tables - ForeachBatch

IsmaelHenzel1
New Contributor

I am wondering how to create complex streaming queries using Delta Live Tables (DLT). I can't find a way to use foreachBatch with it, and this is causing me some difficulty. I need to create a window using a lag without a time range, which is not possible in structured streaming but is fully achievable using foreachBatch in structured streaming.

Is there a way to overcome this limitation?

Additionally, if I create a materialized view in DLT to achieve the same functionality as foreachBatch in Spark Structured Streaming, will it process only the new available batches automatically based on some change data feed, or will it recompute everything again? If it processes the batches incrementally, are materialized views in some way equivalent to the Spark foreachBatch?

1 REPLY 1

Brahmareddy
Valued Contributor II

Hi @IsmaelHenzel1,

How are you doing today?

As per understanding, Consider using Delta Live Tables (DLT) materialized views to handle complex streaming logic as DLT doesn’t currently support foreachBatch. For windowing with lag, DLT materialized views can help, but without a direct time range in streaming, you may need to adjust your approach. If you create a materialized view, it will process data incrementally, similar to foreachBatch, using the underlying change data feed (CDF). This means it should only compute on new data rather than recomputing the entire dataset, giving you incremental processing similar to the behavior in foreachBatch.

Please give a try and let me know for any queries.

Regards,

Brahma

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group