cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

What exact difference does Auto Loader make?

hello_world
New Contributor III

New to Databricks and here is one thing that confuses me.

Since Spark Streaming is already capable of incremental loading by checkpointing. What difference does it make by enabling Auto Loader?

3 REPLIES 3

Aviral-Bhardwaj
Esteemed Contributor III

it have notification system also ,including incremental data processing

Rishabh-Pandey
Honored Contributor II

When you enable Autoloader , you not need to worry about the incoming files , that when it will come or not , in spark streaming files will be coming continously , but suppose you are not sure about the files that when the fill will be come to the landing to get processed , in that scenario , if you autoloader works

autoloader will send the files automatically to get processed whenever the files comes , if you file have arrived on any particular day , it will automatically send the new files only for the processing .

Rishabh Pandey

Meghala
Valued Contributor II

Auto Loader provides a Structured Streaming source called 

cloudFiles

. Given an input directory path on the cloud file storage, the 

cloudFiles

 source automatically processes new files as they arrive, with the option of also processing existing files in that directory. Auto Loader has support for both Python and SQL in Delta Live Tables.

You can use Auto Loader to process billions of files to migrate or backfill a table. Auto Loader scales to support near real-time ingestion of millions of files per hour.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!