I have found that reducing the number of objects in the landing path (via an archive/cleanup process) is the most reliable fix. Auto Loader's file discovery can bog down in big/"long-lived" landing folders—especially in directory-listing mode—so cleaning or moving processed files keeps discovery fast and avoids odd "no new files" stalls. Databricks now exposes this as a first-class knob: cloudFiles.cleanSource with either MOVE (to an archive path) or DELETE. Microsoft Learn+1

Here's a concise root-cause + hardening checklist you can keep:

• #1: Too many files in landing → incremental listing stalls
• Keep the landing zone small. Enable cloudFiles.cleanSource and point cloudFiles.cleanSource.moveDestination to an archive. You already did this manually; using the built-in option makes it automatic and consistent. (Requires newer DBR—see docs.) Microsoft Learn
• Directory-listing vs. file-notification
• Listing mode is the default and simplest, but it can miss or delay files under certain patterns and large directories. If feasible, switch to file-notification mode (uses storage events + queue) or disable incremental listing per the KB guidance for stubborn cases. Microsoft Learnkb.databricks.com
• Overwrites & checkpoints
• With cloudFiles.allowOverwrites=true, Auto Loader will re-ingest the latest version of a file, but mixed signals (mod times vs. event times) and state in the RocksDB checkpoint can cause non-intuitive behaviour if files are edited in place. Your archive flow reduces these edge cases. Microsoft LearnDatabricks DocumentationLakeFS
• General production hardening
• Follow the production guidelines (Lakeflow/streaming best practices), keep write permissions for source + archive path, avoid filenames starting with _, and monitor the stream state. Microsoft Learn+1Medium

Drop-in config (Azure)

sdf = (spark.readStream.format("cloudFiles")

  .option("cloudFiles.format", "json")

  .option("cloudFiles.allowOverwrites", "true")

  # Auto-archive processed files:

  .option("cloudFiles.cleanSource", "MOVE") # or DELETE

  .option("cloudFiles.cleanSource.moveDestination", "abfss://raw@acnt.dfs.core.windows.net/archive/myfeed/")

  .option("cloudFiles.cleanSource.retentionDuration", "30 days") # default; can be shorter for MOVE

  .option("badRecordsPath", bad_records_path)

  .schema(schema)

  .load(loading_path))

Notes: cleanSource runs cleanup after retention; ensure the service principal has write on both the landing and archive paths. These options are documented in the current Auto Loader options reference. Microsoft Learn

When to prefer file-notification mode

If your landing zone is high-churn or very large, wiring storage events + queue (Azure: Event Grid → Queue) gives more deterministic discovery and less listing overhead. The KB on "fails to pick up new files in directory listing mode" explicitly recommends notification mode or disabling incremental listing in tricky cases. kb.databricks.com