cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

How to trigger workflow job tasks from Autoloader

brickster
New Contributor II

I have configured a File Notification Autoloader that monitors S3 bucket for binary files. I want to integrate autoloader with workflow job so that whenever a file is placed in S3 bucket, the pipeline job notebook tasks can pick-up new file and start processing. How can we integrate autoloader with a workflow job to trigger task execution when a file is detected in S3 bucket? Any pointers will be much appreciable. I referred several links but all of them talks up to setting up autoloader, and not about triggering further tasks.

3 REPLIES 3

odoll
New Contributor II

you can trigger a job upon file arrival now.

Just go to Trigger > Add Trigger , Set Trigger Type to File Arrival and enter the S3 path to monitor.

 Screenshot 2022-11-17 at 3.54.51 PM

373231
New Contributor II

I do not see any such option - Trigger on my workflow/tasks. I only see Schedule. Is there anything specific required apart from Unit catalog enablement for workspace (which I did)

Anonymous
Not applicable

Hi @Saravanan Ponnaiah​ 

Hope everything is going great.

Does @odoll odoll​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

We'd love to hear from you.

Thanks!

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.