cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

File arrival trigger customization

xssdfd
New Contributor II

Hi all.

 

I have workflow which I would like to trigger when new file arrive. Problem is that in my storage account, there are few different types of files. Lets assume that I have big csv file and small xlsx mapping file. I would like to trigger job, only when csv file arrive, ideally with specifc name of file, like trigger job only when new file is named like 'Data_from_production*.csv'.  

Is it possible with autoloader? 

 

Thanks all for help 

2 REPLIES 2

feiyun0112
Contributor III

option pathGlobFilter or fileNamePattern

https://docs.databricks.com/en/ingestion/auto-loader/options.html 

xssdfd
New Contributor II

Thank you very much. Thats what I needed. Just to clarify auto loader further. I do not need running cluster to make autoloader works, right? Having it set up and in active state wont charge me until file finally arrive i.e. it is still good option when file arrive in longer period of time like 1 per week or two weeks. Is my understanding correct? 

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.