Updating records with auto loader
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-01-2024 05:00 AM
I want to ingest JSON files from an S3 bucket into a Databricks table using an autoloader.
A job runs every few hours to write the combined JSON data to the table.
Some records might be updates to existing records, identifiable by a specific key.
I want to update existing records if a new record with the same key arrives.
Can this be done with an autoloader? If not, what is the best approach to achieve this?