Ingesting Non-Incremental Data into Delta
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-19-2024 01:46 PM
Hello,
I have non-incremental data landing in a storage account. This data contains old data from before as well as new data. I would like to avoid doing a complete table deletion and table creation just to upload the data from storage and have an updated table. I want to be able to scan the data file, check where the new data begins, and ingest only the new data and append that to my existing delta table. Is there a method in which I can go about this type of data ingestion?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-19-2024 05:29 PM - edited 04-19-2024 05:29 PM
Well, if you know the conditions to separate new data from old data, then while reading the data in to your dataframe, use filter or where clause to select new data and ingest it in to your delta table.
This is how you can do in general. But if you have a complex use case, then please provide samples of your dataset.

