Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
I have configured a File Notification Autoloader that monitors S3 bucket for binary files. I want to integrate autoloader with workflow job so that whenever a file is placed in S3 bucket, the pipeline job notebook tasks can pick-up new file and start processing. How can we integrate autoloader with a workflow job to trigger task execution when a file is detected in S3 bucket? Any pointers will be much appreciable. I referred several links but all of them talks up to setting up autoloader, and not about triggering further tasks.
I do not see any such option - Trigger on my workflow/tasks. I only see Schedule. Is there anything specific required apart from Unit catalog enablement for workspace (which I did)
Does @odoll odoll response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?
We'd love to hear from you.
Thanks!
Connect with Databricks Users in Your Area
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
