Tracking File Arrivals in Nested Folders Using Databricks File Arrival Trigger

[dhanshri]

Hi Team,
I'm currently exploring a file arrival trigger with Data-bricks, but my data is organized into nested folders representing various sources. For instance:

source1

  |-- file1

       |-- file.csv

  |-- file2

       |-- file.csv

My goal is to determine the specific location within the nested folder structure where a file, such as file.csv, has arrived, along with capturing the file name. This allows me to streamline my workflow by utilizing a single job for multiple files.

Could someone provide guidance on how to achieve this effectively within Data-bricks?

Thank you!

[Kaniz_Fatma]

Hi @dhanshri, 

• You can use file arrival triggers to automatically trigger a run of your Databricks job when new files arrive in your specified storage location.
• This feature is particularly useful when data arrives on an irregular schedule, making scheduled jobs less efficient.
• To add a file arrival trigger to your job:
  • In the sidebar, click Workflows.
  • Locate your job in the Jobs tab and click its name.
  • In the Job details panel on the right, click Add trigger.
  • Select File arrival as the trigger type.
• Since your data is organized into nested folders representing various sources, you’ll need to configure the trigger to monitor these folders.
• Here’s how you can approach it:
  • Set up a trigger at the desired level (e.g., ChildFolder2).
  • Use a lookup activity or a Get Metadata Activity to fetch the file with the name in the corresponding ChildFolder1.
  • Check whether the file has been created in ChildFolder1.
  • Optionally, introduce a delay (e.g., 10-15 minutes) using the Wait activity if you want to check aft...
• Keep in mind that a maximum of fifty jobs can be configured with a file arrival trigger in a Databricks workspace.
• Additionally, a storage location configured for a file arrival trigger can contain only up to 10,000 files. Locations with more files cannot be monitored for new file arrivals
• By implementing these steps, you’ll be able to efficiently track file arrivals within your nested folder structure and streamline your workflow.
• Happy Databricks exploration! 

[adriennn]

@Kaniz_Fatma did a LLM Bot write the above response for you? You link to a Stackoverflow post which uses Azure Data Factory, and you text contains concepts which do not apply to Databricks ("Use a lookup activity or a Get Metadata Activity to fetch the file with the name in the"). Actually, your answer seems to be lifted straight from Satckoverflow, pretty sure it's not allowed by their policies...