cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
MVP Articles
This page brings together externally published articles written by our MVPs. Discover expert perspectives, real-world guidance, and community contributions from leaders across the ecosystem.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Watermark-Based Incremental Ingestion (Lakeflow Connect query-based capture)

Hubert-Dudek
Databricks MVP

What if we want to ingest data incrementally without CDF? than we have new functionality from Databricks "query-based capture", which is nothing less than watermark-based incremental ingestion. It seems to be another Best Practice solution for incremental data loading.

https://databrickster.medium.com/watermark-based-incremental-ingestion-lakeflow-connect-query-based-...
https://www.sunnydata.ai/blog/lakeflow-connect-query-based-capture-incremental-ingestionwatermark.png

 

 


My blog: https://databrickster.medium.com/
0 REPLIES 0