How do I process each new record when using autoloader.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-14-2022 08:56 AM
For instance, I'm ingesting webhook data into a delta table with autoloader and need to run a process for each new record as it arrives.
Labels:
- Labels:
-
Autoloader
-
Delta table
-
Webhook
2 REPLIES 2
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-14-2022 05:18 PM
With autoloader, you can do something like changelog and record data about operations performed on each micro batch - like affected id, I/U/D, timestamp etc..
Then you can make use of this changelog table, and run subsequent processes for each row affected per micro batch..
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-15-2022 06:19 AM
Thank you for the info. I'm just starting out with Databricks.

