Tracking File Arrivals in Nested Folders Using Databricks File Arrival Trigger

[dhanshri]

Hi Team,
I'm currently exploring a file arrival trigger with Data-bricks, but my data is organized into nested folders representing various sources. For instance:

source1

  |-- file1

       |-- file.csv

  |-- file2

       |-- file.csv

My goal is to determine the specific location within the nested folder structure where a file, such as file.csv, has arrived, along with capturing the file name. This allows me to streamline my workflow by utilizing a single job for multiple files.

Could someone provide guidance on how to achieve this effectively within Data-bricks?

Thank you!

[adriennn]

@Retired_mod did a LLM Bot write the above response for you? You link to a Stackoverflow post which uses Azure Data Factory, and you text contains concepts which do not apply to Databricks ("Use a lookup activity or a Get Metadata Activity to fetch the file with the name in the"). Actually, your answer seems to be lifted straight from Satckoverflow, pretty sure it's not allowed by their policies...