topic How do I process each new record when using autoloader. in Data Engineering

How do I process each new record when using autoloader.

Gabriel0007 — Thu, 14 Jul 2022 15:56:46 GMT

For instance, I'm ingesting webhook data into a delta table with autoloader and need to run a process for each new record as it arrives.

Re: How do I process each new record when using autoloader.

AmanSehgal — Fri, 15 Jul 2022 00:18:45 GMT

With autoloader, you can do something like changelog and record data about operations performed on each micro batch - like affected id, I/U/D, timestamp etc..

Then you can make use of this changelog table, and run subsequent processes for each row affected per micro batch..

Re: How do I process each new record when using autoloader.

Gabriel0007 — Fri, 15 Jul 2022 13:19:36 GMT

Thank you for the info. I'm just starting out with Databricks.