- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-23-2021 04:03 PM
The primary benefit with AutoLoader would be the abstraction of checkpointing what data has already been processed successfully vs what needs processing. For your use case the File Notification method would work well - https://docs.databricks.com/spark/latest/structured-streaming/auto-loader.html#file-discovery-modes. In addition there are some other benefits of Auto Loader around schema inference and evolution that you may benefit from as well depending on your use case - https://docs.databricks.com/spark/latest/structured-streaming/auto-loader.html#configuration
And finally there's the future proofing of your pipelines with enhancements to Auto Loader that you can benefit from as well