Ingesting Non-Incremental Data into Delta

[anonymous_567]

Hello,

I have non-incremental data landing in a storage account. This data contains old data from before as well as new data. I would like to avoid doing a complete table deletion and table creation just to upload the data from storage and have an updated table. I want to be able to scan the data file, check where the new data begins, and ingest only the new data and append that to my existing delta table. Is there a method in which I can go about this type of data ingestion?

[AmanSehgal]

Well, if you know the conditions to separate new data from old data, then while reading the data in to your dataframe, use filter or where clause to select new data and ingest it in to your delta table.

This is how you can do in general. But if you have a complex use case, then please provide samples of your dataset.