topic Re: ignoreDeletes' option with Delta Live Table streaming source in Data Engineering

ignoreDeletes' option with Delta Live Table streaming source

Zachary_Higgins — Wed, 01 Jun 2022 20:56:25 GMT

We have a delta streaming source in our delta live table pipelines that may have data deleted from time to time.

The error message is pretty self explanatory:

...from streaming source at version 191. This is currently not supported. If you'd like to ignore deletes, set the option 'ignoreDeletes' to 'true'.

What's not clear is how to set this option. This is what we have now but it's not producing the desired results. The desired result being new data is read and deletes are ignored.

SET pipelines.ignoreDeletes = true;
CREATE OR REFRESH STREAMING LIVE TABLE...

How should this option be set in a delta live table?

Re: ignoreDeletes' option with Delta Live Table streaming source

Zachary_Higgins — Thu, 02 Jun 2022 18:00:58 GMT

Hi - Thanks for the response. Does your suggestion work with Delta live tables when you try it? This seems to produce the same error message when I use the code below:

@dlt.table(
   ...
    })
 
def table_fnc():
    return spark.readStream.format("delta").option("ignoreDeletes", "true").table("tablename")

I'm not worried about duplicates. I just want to stream out the tables current state and append it to a sink in my DLT pipeline. As far as I know, DLT can't just append data from a source unless it's streamed in...

Re: ignoreDeletes' option with Delta Live Table streaming source

Zachary_Higgins — Fri, 10 Jun 2022 20:10:10 GMT

I haven't heard back, but the response above was copy and pasted from here: Table streaming reads and writes | Databricks on AWS

We decided to just move these tables to a true structured stream. We hope that DLT can support simple appends later on.

Re: ignoreDeletes' option with Delta Live Table streaming source

JohnA — Mon, 01 Aug 2022 03:00:30 GMT

@Kaniz Fatma Hi Kaniz, can we please circle around to this? Like @Zachary Higgins , I am unsure how to set the ignoreDeletes or ignoreChanges spark.sql configuration for my Delta Live Table Pipeline defined in SQL.

Thanks

Re: ignoreDeletes' option with Delta Live Table streaming source

7effrey — Fri, 11 Nov 2022 12:34:51 GMT

Databricks, please provide an answer to this. It seems like there is no documentation on how delta live tables support table updates. The ignoreChanges is bound to spark.readstream method which is not made to dlt.read_stream

Re: ignoreDeletes' option with Delta Live Table streaming source

Michael42 — Sun, 18 Dec 2022 21:31:37 GMT

I'd am looking at this as well and would like to understand my options here.

Re: ignoreDeletes' option with Delta Live Table streaming source

Zachary_Higgins — Thu, 02 Feb 2023 19:48:28 GMT

@Kaniz Fatma - Has Databricks found a way to prune unwanted records from a source without requiring the entire sink table be recalculated with DLT?

Re: ignoreDeletes' option with Delta Live Table streaming source

yegorski — Fri, 21 Mar 2025 04:38:03 GMT

We had to delete some records in the destination tables created by the DLT pipeline today and hit this error. What resolved it was using spark.readStream and then those options could be set. Can't set them on `dlt.read_stream`

df = dlt.read_stream("request_params")
    # df = dlt.read_stream("request_params")
    df = spark.readStream.option("skipChangeCommits", "true").option("ignoreDeletes", "true").table("operations.api_logs.request_params")

Re: ignoreDeletes' option with Delta Live Table streaming source

yegorski — Fri, 21 Mar 2025 04:39:30 GMT

https://docs.databricks.com/aws/en/structured-streaming/delta-lake

https://docs.databricks.com/aws/en/dlt/python-ref

Re: ignoreDeletes' option with Delta Live Table streaming source

IanB_Argento — Thu, 26 Jun 2025 10:59:09 GMT

I had this same issue whilst doing some POC work. I was able to overcome it as follows:

Navigate to Workflows | Jobs & pipelines.
Select your pipeline.
Click the drop-down next to the Start button.
Choose "Full refresh all".

That resets it all and fixes the problem caused by deleting rows (in my case, all the rows) from the bronze table (which broke the silver table load in the pipeline, but it's fine after full refresh operation). It might not be these best solution for everyone's needs because it'll force a full reload from scratch and maybe you don't even have all your historic source files/messages available anymore.