06-15-2021 07:49 AM
Want to Know what is Autoloader and what are its advantages
06-17-2021 04:54 PM
https://databricks.com/blog/2020/02/24/introducing-databricks-ingest-easy-data-ingestion-into-delta-... ?
06-17-2021 06:44 PM
To add,
Starting in DBR 8.2, Autoloader supports JSON
Starting in DBR 8.3, Autoloader supports CSV
It also now supports Schema Inference, Schema Evolution and Schema Hints.
https://docs.databricks.com/spark/latest/structured-streaming/auto-loader-schema.html
06-17-2021 06:47 PM
The biggest advantage is the ease with which you can star ingesting data from your Cloud Storage directly into a Delta Table. You can choose Directory Listing mode or File Notification mode, depending on what fits your use case best.
