Hi Everyone,
I have an ingestion script where I use the delta merge to update and append newly incoming data in dataframe format to an existing delta table.
I am experiencing an issue where all the variables that have been used previously loose their data after the execution of delta merge.
I have tried using cache and persist (both on disk and in memory) and they don't yield the expected results.
What might be the cause for such behaviour and how can I overcome it?
Thanks
