Tracking File Arrivals in Nested Folders Using Databricks File Arrival Trigger
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-08-2024 02:01 AM
Hi Team,
I'm currently exploring a file arrival trigger with Data-bricks, but my data is organized into nested folders representing various sources. For instance:
I'm currently exploring a file arrival trigger with Data-bricks, but my data is organized into nested folders representing various sources. For instance:
source1
|-- file1
|-- file.csv
|-- file2
|-- file.csv
My goal is to determine the specific location within the nested folder structure where a file, such as file.csv, has arrived, along with capturing the file name. This allows me to streamline my workflow by utilizing a single job for multiple files.
Could someone provide guidance on how to achieve this effectively within Data-bricks?
Thank you!
