Tracking File Arrivals in Nested Folders Using Databricks File Arrival Trigger

dhanshri — Mon, 08 Apr 2024 09:01:27 GMT

Hi Team,
I'm currently exploring a file arrival trigger with Data-bricks, but my data is organized into nested folders representing various sources. For instance:

source1

|-- file1

|-- file.csv

|-- file2

|-- file.csv

My goal is to determine the specific location within the nested folder structure where a file, such as file.csv, has arrived, along with capturing the file name. This allows me to streamline my workflow by utilizing a single job for multiple files.

Could someone provide guidance on how to achieve this effectively within Data-bricks?

Thank you!

Re: Tracking File Arrivals in Nested Folders Using Databricks File Arrival Trigger

adriennn — Thu, 25 Apr 2024 12:54:37 GMT

@Retired_mod did a LLM Bot write the above response for you? You link to a Stackoverflow post which uses Azure Data Factory, and you text contains concepts which do not apply to Databricks ("Use a lookup activity or a Get Metadata Activity to fetch the file with the name in the"). Actually, your answer seems to be lifted straight from Satckoverflow, pretty sure it's not allowed by their policies...

topic Re: Tracking File Arrivals in Nested Folders Using Databricks File Arrival Trigger in Get Started Discussions

Tracking File Arrivals in Nested Folders Using Databricks File Arrival Trigger

Re: Tracking File Arrivals in Nested Folders Using Databricks File Arrival Trigger