cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Tracking File Arrivals in Nested Folders Using Databricks File Arrival Trigger

dhanshri
New Contributor
Hi Team,
I'm currently exploring a file arrival trigger with Data-bricks, but my data is organized into nested folders representing various sources. For instance:
 
source1
  |-- file1
       |-- file.csv
  |-- file2
       |-- file.csv
   
My goal is to determine the specific location within the nested folder structure where a file, such as file.csv, has arrived, along with capturing the file name. This allows me to streamline my workflow by utilizing a single job for multiple files.
 
Could someone provide guidance on how to achieve this effectively within Data-bricks?
 
Thank you!
1 REPLY 1

adriennn
Contributor II

@Retired_mod did a LLM Bot write the above response for you? You link to a Stackoverflow post which uses Azure Data Factory, and you text contains concepts which do not apply to Databricks ("Use a lookup activity or a Get Metadata Activity to fetch the file with the name in the"). Actually, your answer seems to be lifted straight from Satckoverflow, pretty sure it's not allowed by their policies...

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group