We are using Databricks in combination with Azure platforms, specifically working with Azure Blob Storage (Gen2). We frequently mount Azure containers in the Databricks file system and leverage external locations and volumes for Azure containers.
Our use case involves building several data pipelines in Databricks, and we are currently facing an issue with setting up a file arrival trigger. The goal is to trigger a workflow whenever a new file is dropped into an Azure Blob Storage container (Gen2), and we need to pass the complete file path to the subsequent processor in the workflow.
We would appreciate guidance on how to:
- Set up a file arrival trigger in Databricks for Azure Blob Storage (Gen2).
- Capture the file path that triggered the event and pass it as a parameter to the next task in the pipeline.
Any advice or best practices to solve this issue would be greatly appreciated!
Thank you for your time and assistance.
Best regards,
Baburam Shrestha
