cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Workflow file arrival trigger - does it apply to overwritten files?

mvmiller
New Contributor III

I am exploring the use of the "file arrival" trigger for a workflow for a use case I am working on.  I understand from the documentation that it checks every minute for new files in an external location, then initiates the workflow when it detects a new file. 

What I don't know, when we say a "new file", does that include a new file that goes by the same name as a file that is already in the external location, which then updates that file by overwriting it?  In which case, the "modified at" timestamp would communicate that the file has been altered and be, for the purposes of this use case, "new"?

To give a concrete example, say I have an external location of 15 CSV files which are, the vast majority of the time, static tables.  Occasionally, one or more the CSV files will be updated with new content.  When that happens, the file name remains the same, but the underlying data within the file will be different. In other words, the file will be overwritten.  I want to know if the "file arrival" trigger would activate when this happens to any file at the external location that the trigger is set up to look at.

1 REPLY 1

Rajani
Contributor

Hi @mvmiller 
The  "file arrival" trigger for a workflow considers the name of the file,when the same name file was overwritten the workflow didnt triggerred.

hope I answered your question!

 

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!