Auto Loader provides Python and Scala methods to ingest new data from a folder location into a Delta Lake table by using directory listing or file notifications. Here's a quick video (7:00) on how to use Auto Loader for Databricks on AWS with Databricks Runtime (DBR) 8.3 or higher.

To follow along with the video, import this notebook into your workspace:

https://files.training.databricks.com/lms/aws-onboarding-incremental-etl/incremental-etl.dbc

For more on Auto Loader, visit the docs. Question and ideas? Leave a comment below!