topic Re: File Trigger Not Triggering Multiple Runs in Data Engineering

File Trigger Not Triggering Multiple Runs

Sneeze7432 — Wed, 09 Jul 2025 21:05:34 GMT

I have a job with one task which is to run a notebook. The job run is setup with a File arrival trigger with my blob storage as the location. The trigger works and the job will start when a new file arrives in the location, but it does not run for multiple files.

For example I had three files uploaded at different times. First at 3:57:03, second at 3:57:07, and the last at 3:57:10. Three new files arrived, but only one job was started. Why did three jobs not get queued to run?

Re: File Trigger Not Triggering Multiple Runs

nayan_wylde — Wed, 09 Jul 2025 21:23:37 GMT

Did you overwrite the file with the same name because overwriting an existing file with a file of the same name does not trigger a run.

Re: File Trigger Not Triggering Multiple Runs

nayan_wylde — Wed, 09 Jul 2025 21:25:44 GMT

Check if you have configured this two options.

Re: File Trigger Not Triggering Multiple Runs

szymon_dybczak — Wed, 09 Jul 2025 21:30:27 GMT

Hi @Sneeze7432 ,

I think it could be caused by following option Wait after last change in seconds. According to documentation:

"The time to wait to trigger a run after file arrival. Another file arrival in this period resets the timer. This settings can be used when files arrive in batches, and the whole batch needs to be processed after all files have arrived."

An important thing to keep in mind is that "another file arrival in this period resets the timer". Put differently, if you've continuously arriving files, your Workflow will never start as its execution will be continuously delayed. For that reason this setting should be used only to optimize the batch of processed files.

Re: File Trigger Not Triggering Multiple Runs

Sneeze7432 — Thu, 10 Jul 2025 01:46:50 GMT

No each file had a unique name associated with them.

Re: File Trigger Not Triggering Multiple Runs

Sneeze7432 — Thu, 10 Jul 2025 01:51:59 GMT

They are both set to 00h 00m.

Re: File Trigger Not Triggering Multiple Runs

Sneeze7432 — Thu, 10 Jul 2025 01:52:15 GMT

I have the "Wait after last change" setting set to 00h 00m which I would assume means that immediately after a file drops in the storage location the job run will start. I would also assume that means if I drop multiple files in the same location multiple jobs should start, and based on my concurrency limits some may have to be queued.

Re: File Trigger Not Triggering Multiple Runs

szymon_dybczak — Thu, 10 Jul 2025 07:05:28 GMT

I'm just guessing, because unfortunately we don't have insight into how this was implemented, but it seems to me that the Databricks engineers treat files uploaded within a short time interval as a single batch — most likely for optimization purposes. If a trigger were to be generated every second, it wouldn’t be a very efficient approach.
Even that option is specified in minutes (as if they assume that anything below that would still be treated as a single batch).

Re: File Trigger Not Triggering Multiple Runs

MariuszK — Thu, 10 Jul 2025 07:51:05 GMT

It looks like the trigger process files in batches, which means that each of the files uploaded doesn't create a new instance of a job.

Wait after last change in seconds: The time to wait to trigger a run after file arrival. Another file arrival in this period resets the timer. This setting can be used when files arrive in batches, and the whole batch needs to be processed after all files have arrived.

If you need to process files immediately or separately, you can play with Auto Loader configuration.

Re: File Trigger Not Triggering Multiple Runs

nayan_wylde — Thu, 10 Jul 2025 14:47:12 GMT

@Sneeze7432 you can also try editing the max concurrent runs in the workflow.

Re: File Trigger Not Triggering Multiple Runs

Sneeze7432 — Thu, 10 Jul 2025 18:37:35 GMT

What doesn't make sense is that the notification bar will tell me "3 new files" but only one job runs. So even though they can display the number of new files between checks it will still only do one job?

I don't know, it doesn't seem to be setup very well.

Re: File Trigger Not Triggering Multiple Runs

szymon_dybczak — Fri, 11 Jul 2025 07:05:51 GMT

Maybe some databricks employee will jump in and will shed some light about implementation details. But for me treating really short intervals as one batch is quite reasonable approach to avoid massive amount of triggers.

Re: File Trigger Not Triggering Multiple Runs

Sneeze7432 — Sat, 12 Jul 2025 00:01:36 GMT

That doesn't solve the problem of jobs not queueing. That would actually not be good because I could have multiple jobs writing to the same location and potentially overwriting each other creating inaccurate data.

Re: File Trigger Not Triggering Multiple Runs

Sneeze7432 — Sat, 12 Jul 2025 00:04:28 GMT

Same, I would really appreciate more details around this.

Re: File Trigger Not Triggering Multiple Runs

maddy08 — Wed, 25 Feb 2026 12:52:17 GMT

@Sneeze7432 did you solve ?

File arrival group the files when it executes, I verified this with Databricks team.

you may encounter Multiple source matched error during MERGE operations.

to overcome, It’s better to APPEND only into to Raw/bronze layer, then extract and do MERGE into prep/silver layer.

hope this helps!!