Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-26-2021 11:17 PM
you can land your raw data in a folder with a date/timestamp.
f.e. /raw/sales/2021/10/27/dataset.csv
In your spark program you can program to read only from the path with a certain date (today, yesterday etc).
If you get full extracts every day, this works fine. The downside is that your raw storage has a lot of redundant data (because of the full extracts every day) but with data lakes being so cheap that is not a big issue.
The biggest plus for delta imo is the merge functionality which enables you to go for an incremental scenario.