cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

File arrival trigger customization

CaptainJack
New Contributor III

Hi all.

 

I have workflow which I would like to trigger when new file arrive. Problem is that in my storage account, there are few different types of files. Lets assume that I have big csv file and small xlsx mapping file. I would like to trigger job, only when csv file arrive, ideally with specifc name of file, like trigger job only when new file is named like 'Data_from_production*.csv'.  

Is it possible with autoloader? 

 

Thanks all for help 

2 REPLIES 2

feiyun0112
Honored Contributor

option pathGlobFilter or fileNamePattern

https://docs.databricks.com/en/ingestion/auto-loader/options.html 

Thank you very much. Thats what I needed. Just to clarify auto loader further. I do not need running cluster to make autoloader works, right? Having it set up and in active state wont charge me until file finally arrive i.e. it is still good option when file arrive in longer period of time like 1 per week or two weeks. Is my understanding correct? 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group