What exact difference does Auto Loader make?

[hello_world]

New to Databricks and here is one thing that confuses me.

Since Spark Streaming is already capable of incremental loading by checkpointing. What difference does it make by enabling Auto Loader?

[Aviral-Bhardwaj]

it have notification system also ,including incremental data processing

[Rishabh264]

When you enable Autoloader , you not need to worry about the incoming files , that when it will come or not , in spark streaming files will be coming continously , but suppose you are not sure about the files that when the fill will be come to the landing to get processed , in that scenario , if you autoloader works

autoloader will send the files automatically to get processed whenever the files comes , if you file have arrived on any particular day , it will automatically send the new files only for the processing .

[Meghala]

Auto Loader provides a Structured Streaming source called 

cloudFiles

. Given an input directory path on the cloud file storage, the 

cloudFiles

 source automatically processes new files as they arrive, with the option of also processing existing files in that directory. Auto Loader has support for both Python and SQL in Delta Live Tables.

You can use Auto Loader to process billions of files to migrate or backfill a table. Auto Loader scales to support near real-time ingestion of millions of files per hour.