Dooley
Databricks Employee
Databricks Employee

So from my research, you have a domino affect of an error. Once you apply changes to a table you cannot stream after it - however you have a view and then a table so you see the error in the table and not the view. Now if you have inserts you will not see those errors (thus it works sometimes) but if you have updates you will see those errors.

Option 1: You can do the MAIN_FLOW_MOVES function before the apply changes step and try to structure the pipeline to do all the filtering and processing before the apply changes command.

Option 2: So to do ignoreChanges - it will propagate but you will have to deal with duplication - here is the documentation on that. However, to do ignoreChanges you have to do a spark.readStream which will break up your pipeline because that is not a DLT stream function. So that's why you could potentially do this in two pipelines and stitch them together with Databricks workflows.

Now since you use this feature a lot I would suggest you click on the "feedback" button in your Databricks and ask for this feature to be a high priority and finished quickly so you can stream from a table that has apply changes done to it in one single pipeline. I will do the same internally to try to get the awareness.