Databricks

Zachary_Higgins · 06-01-2022

We have a delta streaming source in our delta live table pipelines that may have data deleted from time to time. The error message is pretty self explanatory:...from streaming source at version 191. This is currently not supported. If you'd like to i...

Zachary_Higgins · 02-02-2023

@Kaniz Fatma - Has Databricks found a way to prune unwanted records from a source without requiring the entire sink table be recalculated with DLT?

Zachary_Higgins · 02-02-2023

There doesn't appear to be a way to enforce a retention policy on source tables when defining a structured stream. Setting the options for "ignoreChanges" and "ignoreDeletes" doesn't seem to have any effect at all. CDC does not fill this role either,...

Zachary_Higgins · 06-10-2022

I haven't heard back, but the response above was copy and pasted from here: Table streaming reads and writes | Databricks on AWSWe decided to just move these tables to a true structured stream. We hope that DLT can support simple appends later on.

Zachary_Higgins · 06-02-2022

Hi - Thanks for the response. Does your suggestion work with Delta live tables when you try it? This seems to produce the same error message when I use the code below:@dlt.table( ... }) def table_fnc(): return spark.readStream.format("de...

Zachary_Higgins · 06-01-2022

Not sure about your partitioning schema, but be careful, as you can run into issues with concurrent updates to the same partition. Concurrency control | Databricks on AWS. In this case, I usually set the trigger to run once, running each merge update...

Databricks

User Stats

User Activity

ignoreDeletes' option with Delta Live Table streaming source

Re: ignoreDeletes' option with Delta Live Table streaming source

Re: Limitation as of now in delta live table

Re: ignoreDeletes' option with Delta Live Table streaming source

Re: ignoreDeletes' option with Delta Live Table streaming source

Re: Multiple streaming sources to the same delta table