Autoloader File Notification Mode not working as expected
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-05-2024 02:37 AM
I am using this given code to read from a source location in ADLS Gen 2 Azure Storage Container.
core_df = (
spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "json")
.option("multiLine", "false")
.option('cloudFiles.allowOverwrites', 'true')
.option("cloudFiles.useNotifications", 'true')
.option("cloudFiles.fetchParallelism", 1)
I am reading files of the format
xxxxx.abc.json
When I am using full refresh I am getting all the files in the given container. But When I try and upload a new file and try to use incrementally I am not getting any new file. Is there any option or argument that can help me deal with this scenario
