File arrival trigger customization
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-23-2024 03:36 AM
Hi all.
I have workflow which I would like to trigger when new file arrive. Problem is that in my storage account, there are few different types of files. Lets assume that I have big csv file and small xlsx mapping file. I would like to trigger job, only when csv file arrive, ideally with specifc name of file, like trigger job only when new file is named like 'Data_from_production*.csv'.
Is it possible with autoloader?
Thanks all for help
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-23-2024 04:36 AM
option pathGlobFilter or fileNamePattern
https://docs.databricks.com/en/ingestion/auto-loader/options.html
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-23-2024 06:00 AM
Thank you very much. Thats what I needed. Just to clarify auto loader further. I do not need running cluster to make autoloader works, right? Having it set up and in active state wont charge me until file finally arrive i.e. it is still good option when file arrive in longer period of time like 1 per week or two weeks. Is my understanding correct?