- 11145 Views
- 10 replies
- 8 kudos
Resolved! Get file information while using "Trigger jobs when new files arrive" https://docs.databricks.com/workflows/jobs/file-arrival-triggers.html
I am currently trying to use this feature of "Trigger jobs when new file arrive" in one of my project. I have an s3 bucket in which files are arriving on random days. So I created a job to and set the trigger to "file arrival" type. And within the no...
- 11145 Views
- 10 replies
- 8 kudos
- 8 kudos
I spent some time configuring a setup similar to this. Unfortunately, there's no simple way to do this. There's only {{job.trigger.file_arrival.location}} parameter, but that is pretty much useless, since it points to the directory that we are watchi...
- 8 kudos
- 2047 Views
- 2 replies
- 12 kudos
Databricks now supports event-driven workloads, especially for loading cloud files from external locations. This means you can save costs and resource...
Databricks now supports event-driven workloads, especially for loading cloud files from external locations. This means you can save costs and resources by triggering your Databricks jobs only when new files arrive in your cloud storage instead of mou...
- 2047 Views
- 2 replies
- 12 kudos
- 12 kudos
Hi @Hubert Dudek​ We really appreciate you sharing this bit of information.Cheers!
- 12 kudos