- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-27-2025 01:54 AM - edited 09-27-2025 01:55 AM
Hello @saurabh18cs !
You don’t need to choose queues and simply use the “File events” path. When enabled, Databricks uses one managed queue per external location (Unity Catalogue), and all your streams that read from that location share it. This avoids cloud queue limits, and Databricks auto-tunes the notification plumbing.
Doc: https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/file-notification-mode...
If you stay in the current file notification mode, the default pattern is one queue per stream, and Databricks recommends fan-out only when you’re constrained by provider limits
I think throughput has no fixed per-second cap. By default, it processes up to 1,000 files per micro-batch; tune with cloudFiles.maxFilesPerTrigger
Please let me know if you have any further questions.