Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
I have configured a File Notification Autoloader that monitors S3 bucket for binary files. I want to integrate autoloader with workflow job so that whenever a file is placed in S3 bucket, the pipeline job notebook tasks can pick-up new file and start processing. How can we integrate autoloader with a workflow job to trigger task execution when a file is detected in S3 bucket? Any pointers will be much appreciable. I referred several links but all of them talks up to setting up autoloader, and not about triggering further tasks.
I do not see any such option - Trigger on my workflow/tasks. I only see Schedule. Is there anything specific required apart from Unit catalog enablement for workspace (which I did)
Does @odoll odoll response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?
