topic Re: File information is not passed to trigger job on file arrival in Data Engineering

File information is not passed to trigger job on file arrival

Rik — Mon, 07 Aug 2023 14:57:18 GMT

We are using the UC mechanism for triggering jobs on file arrival, as described here: https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/file-arrival-triggers.

Unfortunately, the trigger doesn't actually pass the file-path that is generating the trigger to the job... (The Run Parameters are empty). Is there any way to get this information?

Re: File information is not passed to trigger job on file arrival

Tharun-Kumar — Tue, 08 Aug 2023 05:03:44 GMT

@Rik

For now, we do not send the file details as part of the trigger. The trigger is used to run a pipeline.

Alternately, You can use autoloader as part of the triggered pipeline to get the details of the file that arrived.

Re: File information is not passed to trigger job on file arrival

Rik — Tue, 08 Aug 2023 12:36:18 GMT

"Alternately, You can use autoloader as part of the triggered pipeline to get the details of the file that arrived."

That doesn't quite fit our requirements unfortunately... Are there any plans on adding this functionality?

Re: File information is not passed to trigger job on file arrival

Tharun-Kumar — Tue, 08 Aug 2023 17:48:46 GMT

@Rik

We have got this request from other Customers too. Our Engineering team is already notified of this and there is an internal ticket for the same. But we don't have an ETA for now.

Re: File information is not passed to trigger job on file arrival

Panda — Thu, 15 Feb 2024 18:24:57 GMT

Could you please provide an update on the status of this particular request? Additionally, do we have any ETA for it?

Re: File information is not passed to trigger job on file arrival

marcuskw — Wed, 05 Jun 2024 08:18:23 GMT

Also something I'm interested in using, would be really helpful to use File Trigger and get relevant information about exactly what file triggered the workflow!

Re: File information is not passed to trigger job on file arrival

srsnarendran — Mon, 23 Sep 2024 16:22:13 GMT

Any ETA ? We are having to use other Orchestration products because of this limitation.

Re: File information is not passed to trigger job on file arrival

artemich — Mon, 23 Sep 2024 19:37:46 GMT

Same here!

Additionally would be great to enhance it to support not just the path to a directory, but also the prefix of the file name (or regex for bonus points). Right now if you have 10 types of files arriving to the same folder, it would be much cleaner to have each workflow handling a given type only process the relevant file arrived.

Re: File information is not passed to trigger job on file arrival

marcuskw — Mon, 23 Sep 2024 19:46:01 GMT

You are able to provide filter options to select only relevant files:
https://docs.databricks.com/en/ingestion/cloud-object-storage/auto-loader/patterns.html#filtering-directories-or-files-using-glob-patterns

Re: File information is not passed to trigger job on file arrival

artemich — Tue, 24 Sep 2024 19:38:36 GMT

For loading file with AutoLoader - for sure. My wish is to have similar capability for File Arrival Trigger.

"A file arrival trigger can be configured to monitor the root of a Unity Catalog external location or volume, or a subpath of an external location or volume."
https://learn.microsoft.com/en-us/azure/databricks/jobs/file-arrival-triggers

Quite often files for multiple data entities (or even pipelines) land in the same directories from a given provider and it would be great to be able to easily manage such scenarios.

Re: File information is not passed to trigger job on file arrival

mattiazeni — Wed, 23 Oct 2024 09:31:55 GMT

What do you need to achieve?

Autoloader is much more efficient since it can handle a bunch of files (only new ones) in a single operation. Handling file by file, especially with a lot of files, will increase latency and increase costs.

Re: File information is not passed to trigger job on file arrival

marcuskw — Wed, 23 Oct 2024 13:24:48 GMT

What I wanted to achieve was a dynamic schema application based on what file was picked up.
So I implement 1 autoloader task to collect files from a specific path "source":
- source/employees/0001.csv
- source/holiday/0001.csv

If the path of the file was available I could then apply the relevant schema in runtime.
But autoloader may want to process both files and put them into the same dataframe?
Maybe this isn't the best usecase, I guess you would recommend to implement multiple tasks/checkpoints for the respective folders?

Re: File information is not passed to trigger job on file arrival

007 — Tue, 18 Mar 2025 10:57:27 GMT

any news?

Re: File information is not passed to trigger job on file arrival

Panda — Thu, 20 Mar 2025 20:05:25 GMT

@007 - Review the link https://community.databricks.com/t5/data-engineering/file-arrival-trigger/m-p/94069/highlight/true#M38808