https://community.databricks.com/t5/technical-blog/incremental-replace-where-flows-brings-targeted-refreshes-to-sdp/ba-p/159057 <P><SPAN>Lakeflow Spark Declarative Pipelines (SDP) has long supported two ways to maintain a table: a </SPAN><STRONG>streaming flow</STRONG><SPAN> for incremental ingestion</SPAN><SPAN>&nbsp;and a&nbsp;<STRONG>materialized view</STRONG>&nbsp;for a fully derived</SPAN><SPAN>&nbsp;result that is always exact. Both are great fits for the shapes they were designed for. What's been missing is a third pattern, the one many batch ETL teams already work in: </SPAN><I><SPAN>"on this scheduled run, refresh just the last 7 days; leave the rest of the table alone."</SPAN></I><I></I></P> <P><SPAN>That's what incremental <CODE>REPLACE WHERE</CODE></SPAN><SPAN>&nbsp;flows bring to SDP and DBSQL — a third option, a </SPAN><STRONG>batch processing flow</STRONG><SPAN> with automatic incrementalization. You declare the slice each run owns with a single predicate, and the pipeline refreshes exactly that slice on every update. The same primitive also supports a one-time predicate override, so the occasional fix-a-column-and-backfill case uses the same mechanism, with no need to recompute the full table.&nbsp;</SPAN></P> <P><SPAN>And it's more than the <CODE>REPLACE WHERE</CODE></SPAN><SPAN>&nbsp;you'd write in Databricks SQL. Incremental <CODE>REPLACE WHERE</CODE></SPAN><SPAN>&nbsp;flows are powered by </SPAN><STRONG>Enzyme</STRONG><SPAN>, Databricks' incremental computation engine: instead of overwriting all data that matches the predicate, the engine only rewrites the files with rows that changed since the last update. With Enzyme, refreshes are both faster and cheaper than a naive partition overwrite – over </SPAN><STRONG>3.4x </STRONG><SPAN>faster and</SPAN><STRONG> 2.5x </STRONG><SPAN>cheaper in an incremental TPC-DI benchmark.&nbsp;&nbsp;</SPAN></P> <P><SPAN><CODE>REPLACE WHERE</CODE> is available now in Beta — and it is available in both Spark Lakeflow Spark Declarative Pipelines and </SPAN><SPAN>DBSQL. You can try it today in the</SPAN><SPAN> Databricks SQL Query Editor, running directly on your SQL warehouse.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="fran_martin_0-1781538236592.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/27815i11FCD1C53F0766A6/image-size/large?v=v2&amp;px=999" role="button" title="fran_martin_0-1781538236592.png" alt="fran_martin_0-1781538236592.png" /></span></P> <P>&nbsp;</P> <H2><STRONG>The problems incremental REPLACE WHERE solves</STRONG></H2> <P><SPAN>The standing predicate — a condition baked into the pipeline definition that tells every scheduled run </SPAN><I><SPAN>"this is the slice you own"</SPAN></I><SPAN> — is the everyday case. It solves a few types of problems naturally, all the same shape: a slice you can name upfront that is the same every run.</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><STRONG>Rolling refresh.</STRONG><SPAN> A source continuously adds new rows and occasionally corrects older ones. You want each nightly run to refresh the last 7 days so corrections or late-arriving data get absorbed, and everything older — which is stable — left untouched. A standing <CODE>REPLACE_WHERE&nbsp;order_date &gt;= date_add(current_date(), -7)</CODE></SPAN><SPAN>&nbsp;becomes the pipeline's contract: every run rebuilds the trailing 7-day window and leaves the rest of the table as-is.</SPAN></LI> </UL> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="fran_martin_1-1781531405652.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/27809iFFAD907CC0342A6D/image-size/large?v=v2&amp;px=999" role="button" title="fran_martin_1-1781531405652.png" alt="fran_martin_1-1781531405652.png" /></span></P> <UL> <LI style="font-weight: 400;" aria-level="1"><STRONG>Building lasting tables from short-retention sources.</STRONG><SPAN> Your raw events table keeps only a few days of data, while the daily aggregates you derive from it need to live permanently. A standing <CODE>REPLACE_WHERE&nbsp;event_date &gt;= date_add(current_date(), -3)</CODE></SPAN><SPAN>&nbsp;</SPAN><SPAN>&nbsp;rebuilds the recent three days each run, and older days persist. A Materialized View would remove rows where the event date is older than 3, since the source no longer contains them.&nbsp;</SPAN></LI> </UL> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="fran_martin_2-1781531405653.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/27810i5810A4C30D0C4107/image-size/large?v=v2&amp;px=999" role="button" title="fran_martin_2-1781531405653.png" alt="fran_martin_2-1781531405653.png" /></span></P> <UL> <LI style="font-weight: 400;" aria-level="1"><STRONG>Scoping dimension updates to recent facts only.</STRONG><SPAN> You enrich a fact table by joining a dimension and writing the dimension value into the fact row at processing time. When the dimension changes later (a user moves regions, a product gets reclassified), you don't want every historical fact row rewritten — purely a </SPAN><STRONG>cost</STRONG><SPAN> decision: rewriting five years of facts every time a dimension row updates is expensive. A standing <CODE>REPLACE_WHERE&nbsp;order_date &gt;= date_add(current_date(), -3)</CODE></SPAN><SPAN>&nbsp;</SPAN><SPAN>&nbsp;scopes each refresh to the recent window, so only recent facts pick up the current dimension values, while older facts retain the values they were enriched with originally.</SPAN></LI> </UL> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="fran_martin_3-1781531405653.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/27808iC89CEC049033F03E/image-size/large?v=v2&amp;px=999" role="button" title="fran_martin_3-1781531405653.png" alt="fran_martin_3-1781531405653.png" /></span></P> <P><SPAN>What these share: the slice each run owns is knowable up front and the same every run. You declare it once in the pipeline definition, and every scheduled update honors it.</SPAN></P> <H2>&nbsp;</H2> <H2><STRONG>Deep dive into REPLACE WHERE</STRONG></H2> <H3><STRONG>1. Syntax overview</STRONG></H3> <P><SPAN>The syntax to use <CODE>REPLACE WHERE</CODE></SPAN><SPAN>&nbsp;is simple; the </SPAN><STRONG>predicate is the table's contract, not part of the <SPAN><CODE>SELECT</CODE></SPAN>.</STRONG><SPAN> You declare which slice this table owns once, at the top of the flow. SDP applies the predicate automatically — don't repeat it inside the body, and don't filter the source; SDP only reads the rows that match.</SPAN></P> <LI-CODE lang="markup">CREATE STREAMING TABLE silver_orders FLOW REPLACE WHERE order_date &gt;= date_add(current_date(), -7) BY NAME SELECT order_id, order_date, region, product_id, customer_id, quantity, unit_price, CAST(quantity * unit_price AS DECIMAL(12,2)) AS revenue, _ingested_at FROM bronze_orders;</LI-CODE> <P><SPAN>To make this concrete, here's a self-contained three-layer example that you can copy and paste into a pipeline:&nbsp;</SPAN></P> <P><STRONG><I>Source Data Generation:</I></STRONG> <I><SPAN>Run it into the SQL editor to generate source data</SPAN></I></P> <LI-CODE lang="markup">CREATE SCHEMA IF NOT EXISTS &lt;catalog&gt;.&lt;schema&gt;; USE CATALOG &lt;catalog&gt;; USE SCHEMA &lt;schema&gt;; CREATE TABLE IF NOT EXISTS bronze_orders ( order_id STRING, order_date DATE, customer_id STRING, product_id STRING, region STRING, customer_segment STRING, customer_name STRING, product_name STRING, product_category STRING, list_price DECIMAL(10,2), quantity INT, unit_price DECIMAL(10,2), event_ts TIMESTAMP, _seq BIGINT, _ingested_at TIMESTAMP ) CLUSTER BY (order_date) TBLPROPERTIES (delta.enableRowTracking = true); INSERT INTO bronze_orders SELECT concat('ORD-', id), date_sub(current_date(), CAST(id % 90 AS INT)), concat('CUST-', lpad(CAST(id % 500 AS STRING), 5, '0')), concat('SKU-', lpad(CAST(id % 25 AS STRING), 3, '0')), element_at(array('AMER','EMEA','APAC','LATAM'), CAST(id % 4 AS INT) + 1), element_at(array('Bronze','Silver','Gold','Platinum'), CAST(((id % 4) + CAST(id / 5000 AS INT)) % 4 AS INT) + 1), concat('Customer ', CAST(id % 500 AS INT)), concat('Product ', CAST(id % 25 AS INT)), element_at(array('Electronics','Apparel','Home','Grocery','Sports'), CAST(id % 5 AS INT) + 1), CAST(20 + (((id % 25) + CAST(id / 5000 AS INT)) % 10) * 5 AS DECIMAL(10,2)), CAST((id % 9) + 1 AS INT), CAST(10 + (id % 190) AS DECIMAL(10,2)), current_timestamp(), id, current_timestamp() FROM range(0, 2000);</LI-CODE> <P><SPAN><STRONG><I>02_silver.sql:</I></STRONG> <I>Replace the last 7 days only</I></SPAN></P> <LI-CODE lang="markup">CREATE STREAMING TABLE silver_orders CLUSTER BY (order_date) FLOW REPLACE WHERE order_date &gt;= date_add(current_date(), -7) BY NAME SELECT order_id, order_date, customer_id, product_id, region, quantity, unit_price, CAST(quantity * unit_price AS DECIMAL(12,2)) AS revenue, event_ts, _ingested_at FROM bronze_orders; </LI-CODE> <P><STRONG><I>03_gold.sql:</I></STRONG><I><SPAN> Combines <CODE>AUTO CDC</CODE>&nbsp;for dimensions and <CODE>REPLACE WHERE</CODE>&nbsp;for the fact table</SPAN></I></P> <LI-CODE lang="markup">CREATE OR REFRESH STREAMING TABLE dim_customer COMMENT "SCD Type 1 customer dimension (AUTO CDC)." FLOW AUTO CDC FROM STREAM bronze_orders KEYS (customer_id) SEQUENCE BY _seq COLUMNS customer_id, region, customer_segment, customer_name STORED AS SCD TYPE 1; CREATE OR REFRESH STREAMING TABLE dim_product COMMENT "SCD Type 1 product dimension (AUTO CDC)." FLOW AUTO CDC FROM STREAM bronze_orders KEYS (product_id) SEQUENCE BY _seq COLUMNS product_id, product_name, product_category, list_price STORED AS SCD TYPE 1; CREATE STREAMING TABLE daily_sales CLUSTER BY (order_date) FLOW REPLACE WHERE order_date &gt;= date_add(current_date(), -7) BY NAME SELECT order_date, customer_id, product_id, COUNT(*) AS order_count, SUM(quantity) AS units_sold, CAST(SUM(revenue) AS DECIMAL(18,2)) AS revenue_usd FROM silver_orders GROUP BY ALL;</LI-CODE> <P><SPAN>As you can see, we created a</SPAN><STRONG> bronze table </STRONG><SPAN>that seeds orders inline (no external source needed), a</SPAN><STRONG> silver streaming table</STRONG><SPAN> fed by a <CODE>REPLACE WHERE</CODE></SPAN><SPAN>&nbsp;flow that refreshes only the last 7 days</SPAN> <SPAN>on every update, and the downstream </SPAN><STRONG>gold dimensions and fact table</STRONG><SPAN>. In the gold layer, you can also see how <CODE>AUTO CDC</CODE></SPAN><SPAN>&nbsp; and <CODE>REPLACE WHERE</CODE></SPAN><SPAN>&nbsp;both can be used in the same pipeline</SPAN> <SPAN>for the creation of the dimensions and the fact table.&nbsp;</SPAN></P> <P><SPAN>You can learn more about <CODE>AUTO CDC</CODE></SPAN><SPAN>&nbsp;in our previous blogpost:&nbsp;</SPAN><A href="https://community.databricks.com/t5/technical-blog/from-150-lines-of-merge-into-to-7-lines-of-sql-auto-cdc-comes-to/ba-p/155355" target="_blank" rel="noopener"><SPAN>https://community.databricks.com/t5/technical-blog/from-150-lines-of-merge-into-to-7-lines-of-sql-auto-cdc-comes-to/ba-p/155355</SPAN></A></P> <H3>&nbsp;</H3> <H3><STRONG>2. The magic under the hood: the Enzyme optimizer</STRONG></H3> <P><SPAN>This is what makes a <CODE>REPLACE WHERE</CODE></SPAN><SPAN>&nbsp;flow inside SDP different from the same operation written by hand in Spark.</SPAN></P> <P><STRONG>First, the baseline.</STRONG><SPAN> Every <CODE>REPLACE WHERE</CODE></SPAN><SPAN>&nbsp;update commits as a single Delta transaction that, mechanically: (1) deletes the rows matching the predicate, (2) recomputes the source query for the predicate range — the predicate is pushed down to the source read, so rows outside the window are never scanned — and (3) inserts the recomputed rows. That alone gives you two guarantees for free: the update is </SPAN><STRONG>atomic and recoverable</STRONG><SPAN> (readers never see a partial state; a failed run leaves the last good state intact), and it is </SPAN><STRONG>predicate-scoped</STRONG><SPAN> (rows outside the window are never touched — no file rewrite, no version churn).</SPAN></P> <P><STRONG>Now compare this to doing it yourself in DBSQL.</STRONG><SPAN> Outside SDP, you would do something like this:</SPAN></P> <LI-CODE lang="markup">INSERT INTO TABLE events REPLACE WHERE start_date &gt;= (NOW() - INTERVAL 7 DAY) SELECT * FROM replace_data</LI-CODE> <P><SPAN>This statement </SPAN><STRONG>always rewrites everything that matches the predicate</STRONG><SPAN>, on every run, whether or not the underlying data changed. Tell it "the last 7 days," and it recomputes and rewrites all seven days, full stop — full source recompute over the window plus a full file rewrite. Dynamic partition overwrite also only works at </SPAN><I><SPAN>partition</SPAN></I><SPAN> granularity and requires the table to be physically partitioned the right way.</SPAN></P> <P><STRONG>This is where Enzyme comes in.</STRONG><SPAN> The engine doesn't stop at "overwrite everything that matches." Enzyme inspects the data matching the predicate, recognizes that most of it hasn't changed since the last run, and only rewrites the files that contain rows that actually changed. Same one-line predicate, a fraction of the I/O.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="fran_martin_4-1781531405654.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/27811i4295E441D4486E7D/image-size/large?v=v2&amp;px=999" role="button" title="fran_martin_4-1781531405654.png" alt="fran_martin_4-1781531405654.png" /></span></P> <P><STRONG>And it’s a lot faster and cheaper too. </STRONG><SPAN>On an incremental TPC-DI benchmark, the incremental <CODE>REPLACE WHERE</CODE> ran </SPAN><STRONG>3.4x </STRONG><SPAN>faster and</SPAN><STRONG> 2.5x </STRONG><SPAN>cheaper than a full <CODE>REPLACE WHERE</CODE></SPAN><SPAN>, with identical output.</SPAN></P> <P><SPAN>The difference is what happens inside the predicate. A regular <CODE>REPLACE WHERE</CODE></SPAN><SPAN>&nbsp;overwrites everything that matches the predicate, so a 7-day window means all seven days get rewritten on every run, even the rows that didn't change. The incremental version scopes the work further: within that same window, it rewrites </SPAN><STRONG>only the rows that actually changed</STRONG><SPAN> and leaves the rest in place. You get the same result, but you stop paying to overwrite data that was already correct.</SPAN></P> <P><STRONG>When incremental processing engages.</STRONG><SPAN> Enzyme isn't guaranteed on every run — and that's fine, the output is always correct either way; the only difference is how much work it takes. It engages when all of these hold: the </SPAN><STRONG>query shape is supported</STRONG><SPAN>; the predicate references </SPAN><STRONG>base columns</STRONG><SPAN> (Predicates on derived values, such as aggregate or window function outputs disables incremental refresh); the predicate is </SPAN><STRONG>deterministic</STRONG><SPAN> (temporal functions such as </SPAN><SPAN>current_date()</SPAN><SPAN> are allowed, </SPAN><SPAN>rand()</SPAN><SPAN> is not); </SPAN><STRONG>no external DML</STRONG><SPAN> modified the window since the last update; and the window doesn't extend into rows the previous predicate excluded. About the type of computation to use, it will just work</SPAN><STRONG> if you use DBSQL, </STRONG><SPAN>but make sure you use </SPAN><STRONG>serverless</STRONG><SPAN> compute for </SPAN><STRONG>Spark Declarative Pipelines</STRONG><SPAN>. Finally, it is worth mentioning that the first refresh is always a full computation; incremental begins on the second update.</SPAN></P> <P><STRONG>Diagnosing a fallback.</STRONG><SPAN> If a condition isn't met, the engine falls back to a full recompute of the window (correct, just heavier) and records why in the pipeline event log's </SPAN><SPAN>planning_information</SPAN><SPAN>:</SPAN></P> <TABLE> <TBODY> <TR> <TD> <P><STRONG>Reason code</STRONG></P> </TD> <TD> <P><STRONG>What happened</STRONG></P> </TD> </TR> <TR> <TD> <P><SPAN>EXTERNAL_CHANGE_IN_REPLACE_WINDOW</SPAN></P> </TD> <TD> <P><SPAN>Something outside the pipeline modified rows inside the window since the last update.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>REPLACE_WHERE_NOT_DETERMINISTIC</SPAN></P> </TD> <TD> <P><SPAN>The current predicate uses non-deterministic expressions.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>PRIOR_REPLACE_WHERE_NOT_DETERMINISTIC</SPAN></P> </TD> <TD> <P><SPAN>The previous run's predicate wasn't deterministic.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>UNSUPPORTED_REPLACE_WHERE_PREDICATE</SPAN></P> </TD> <TD> <P><SPAN>The query shape or predicate isn't on the fast path (or this was a predicate override).</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>PRIOR_REPLACE_WHERE_PREDICATE_MISSING</SPAN></P> </TD> <TD> <P><SPAN>The previous predicate is missing from the provenance.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>STALE_TARGET_DATA_IN_WINDOW</SPAN></P> </TD> <TD> <P><SPAN>The replace where predicated changed across updates, and the new window includes rows that were excluded by the previous predicate.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>CANNOT_PUSH_PREDICATE</SPAN></P> </TD> <TD> <P><SPAN>The replace where predicate could not be pushed down to at least one source.</SPAN></P> </TD> </TR> <TR> <TD> <P><SPAN>REPLACE_WHERE_BACKFILL</SPAN></P> </TD> <TD> <P><SPAN>The update is a backfill — a one-off predicate override used to populate historical data.</SPAN></P> </TD> </TR> </TBODY> </TABLE> <P><SPAN>To stay on the fast path: predicate on base columns with a moving lower bound; include the predicate column in the <CODE>GROUP BY</CODE></SPAN><SPAN>&nbsp;for aggregations and in the join key for joins; and don't write to the target from outside the pipeline inside the replace window.</SPAN></P> <H2>&nbsp;</H2> <H2><STRONG>The predicate override: a one-time update for backfills and corrections</STRONG></H2> <P><SPAN>Most of the time, the pipeline runs normally, but occasionally a single update needs a different scope. A few common examples:</SPAN></P> <UL> <LI aria-level="1"> <P><STRONG>Targeted corrections. </STRONG><SPAN>A conversion bug caused revenue for <CODE>country = 'CA'</CODE>&nbsp;to be incorrect over</SPAN><SPAN>&nbsp;the last 30 days. The standing 7-day predicate won't reach back that far, and you don't want to recompute everything else. A one-time update can rewrite just <CODE>country = 'CA'</CODE></SPAN><SPAN>&nbsp;for that window, then the pipeline returns to its standing predicate.</SPAN></P> </LI> </UL> <P><LI-WRAPPER></LI-WRAPPER></P> <UL> <LI style="font-weight: 400;" aria-level="1"> <P><STRONG>Backfills: </STRONG><SPAN>You add a new metric to a daily aggregate or change how a column handles an edge case. From now on, every run uses the new logic — but you also want it applied to a chosen historical range. Or: you keep the standing window small while validating the transformation, then load the full history once you're confident.</SPAN></P> </LI> </UL> <P><SPAN>Both are </SPAN><STRONG>predicate overrides</STRONG><SPAN> — a one-time run where the standing predicate is swapped for a different expression, for that update only. The pipeline definition remains untouched; the next scheduled run reverts to the standing predicate.</SPAN></P> <P><STRONG>In Spark Declarative Pipelines, </STRONG><SPAN>the override currently runs through the API. The docs include a helper function with examples:&nbsp;</SPAN><A href="https://docs.databricks.com/aws/en/ldp/flows-replace-where#backfill-historical-data" target="_blank" rel="noopener"><SPAN>https://docs.databricks.com/aws/en/ldp/flows-replace-where#backfill-historical-data</SPAN></A></P> <P><SPAN><STRONG>In DBSQL, <CODE>REFRESH</CODE></STRONG>&nbsp;overrides are coming soon; in the meantime,&nbsp;<STRONG>arbitrary DML&nbsp;</STRONG>is supported: you can <CODE>INSERT</CODE>, <CODE>SELECT</CODE>, or <CODE>DELETE</CODE> against the table from&nbsp;<EM>outside</EM>&nbsp;the pipeline, like for any other streaming table.</SPAN></P> <H2>&nbsp;</H2> <H2><STRONG>How to get started</STRONG></H2> <P><SPAN><CODE>REPLACE WHERE</CODE>&nbsp;</SPAN><SPAN>flows are in </SPAN><STRONG>Beta</STRONG><SPAN>. To wire one up: set your pipeline channel to </SPAN><SPAN>PREVIEW</SPAN><SPAN>, use Unity Catalog, and run on </SPAN><STRONG>DBSQL</STRONG><SPAN> or </SPAN><STRONG>serverless compute</STRONG><SPAN> (required for incremental processing via Enzyme).</SPAN></P> <P><SPAN>A good way to start is to take the self-contained sample provided above, copy it to a new Spark Declartive Pipeline, and run it.&nbsp;</SPAN></P> <P><STRONG>A few limitations to know:</STRONG><SPAN> A table can have only one <CODE>REPLACE WHERE</CODE></SPAN><SPAN>&nbsp;flow (and can't also be an <CODE>AUTO CDC</CODE></SPAN><SPAN>&nbsp;or append target); expectations aren't supported on <CODE>REPLACE WHERE</CODE></SPAN><SPAN>&nbsp;targets (set them upstream); and <CODE>FULL REFRESH</CODE></SPAN><SPAN>&nbsp;is destructive — it re-runs the flow with its </SPAN><I><SPAN>standing</SPAN></I><SPAN> predicate, so any backfilled history is lost (set </SPAN><SPAN>pipelines.reset.allowed = false</SPAN><SPAN> to block it).</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><STRONG>REPLACE WHERE in DBSQL </STRONG><SPAN><A href="https://docs.databricks.com/aws/en/ldp/dbsql/flows-replace-where" target="_blank" rel="noopener">https://docs.databricks.com/aws/en/ldp/dbsql/flows-replace-where</A></SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>REPLACE WHERE flows for Lakeflow Declarative Pipelines</STRONG><SPAN> — </SPAN><A href="https://docs.databricks.com/aws/en/ldp/flows-replace-where" target="_blank" rel="noopener"><SPAN>https://docs.databricks.com/aws/en/ldp/flows-replace-where</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Backfilling historical data</STRONG><SPAN> — </SPAN><A href="https://docs.databricks.com/aws/en/ldp/flows-replace-where#backfill-historical-data" target="_blank" rel="noopener"><SPAN>https://docs.databricks.com/aws/en/ldp/flows-replace-where#backfill-historical-data</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Incremental refresh &amp; supported query shapes</STRONG><SPAN> — </SPAN><A href="https://docs.databricks.com/aws/en/ldp/flows-replace-where#best-practices-for-incremental-refresh" target="_blank" rel="noopener"><SPAN>https://docs.databricks.com/aws/en/ldp/flows-replace-where#best-practices-for-incremental-refresh</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Limitations </STRONG><SPAN>— </SPAN><A href="https://docs.databricks.com/aws/en/ldp/flows-replace-where#limitations" target="_blank" rel="noopener"><SPAN>https://docs.databricks.com/aws/en/ldp/flows-replace-where#limitations</SPAN></A></LI> </UL> <P><SPAN>If you have feedback or questions, drop them in the comments below.</SPAN></P> Tue, 16 Jun 2026 16:44:13 GMT fran_martin 2026-06-16T16:44:13Z https://community.databricks.com/t5/technical-blog/incremental-replace-where-flows-brings-targeted-refreshes-to-sdp/ba-p/159057 <P>SDP has always given you two ways to keep a table fresh: a streaming flow for incremental ingestion, or a materialized view that recomputes an exact result. But most batch ETL teams live in a third world entirely: "refresh just the last 7 days, leave everything else alone." Incremental <SPAN><CODE>REPLACE WHERE</CODE></SPAN> flows finally make that a first-class pattern, with Enzyme doing the incremental math, so refreshes run over 3.4x faster and 2.5x cheaper than a naive partition overwrite.</P> Tue, 16 Jun 2026 16:44:13 GMT https://community.databricks.com/t5/technical-blog/incremental-replace-where-flows-brings-targeted-refreshes-to-sdp/ba-p/159057 fran_martin 2026-06-16T16:44:13Z