Hi Everyone,

I have an ingestion script where I use the delta merge to update and append newly incoming data in dataframe format to an existing delta table.

I am experiencing an issue where all the variables that have been used previously loose their data after the execution of delta merge.

I have tried using cache and persist (both on disk and in memory) and they don't yield the expected results.

What might be the cause for such behaviour and how can I overcome it?

Thanks