topic File arrival trigger customization in Data Engineering

File arrival trigger customization

CaptainJack — Fri, 23 Feb 2024 11:36:21 GMT

Hi all.

I have workflow which I would like to trigger when new file arrive. Problem is that in my storage account, there are few different types of files. Lets assume that I have big csv file and small xlsx mapping file. I would like to trigger job, only when csv file arrive, ideally with specifc name of file, like trigger job only when new file is named like 'Data_from_production*.csv'.

Is it possible with autoloader?

Thanks all for help

Re: File arrival trigger customization

feiyun0112 — Fri, 23 Feb 2024 12:36:21 GMT

option pathGlobFilter or fileNamePattern

https://docs.databricks.com/en/ingestion/auto-loader/options.html

Re: File arrival trigger customization

CaptainJack — Fri, 23 Feb 2024 14:00:36 GMT

Thank you very much. Thats what I needed. Just to clarify auto loader further. I do not need running cluster to make autoloader works, right? Having it set up and in active state wont charge me until file finally arrive i.e. it is still good option when file arrive in longer period of time like 1 per week or two weeks. Is my understanding correct?