Workflow file watch - capture filename trigger

csmcpherson — Thu, 05 Sep 2024 22:35:53 GMT

With respect to the file watch trigger in workflows, how can we capture what files and or path was identified as raising the trigger?

I'd like to use this information to set parameters based upon the file name and the file path

Thank you!

https://docs.databricks.com/en/jobs/file-arrival-triggers.html

Re: Workflow file watch - capture filename trigger

szymon_dybczak — Fri, 06 Sep 2024 09:50:51 GMT

Hi @csmcpherson ,

This is currently not supported, but databricks team is working on that idea according to below thread:

Solved: File information is not passed to trigger job on f... - Databricks Community - 39266

As a workaround, if you use autoloader, you can use file _metadata column.

File metadata column - Azure Databricks | Microsoft Learn

spark.readStream \ .format("cloudFiles") \ .option("cloudFiles.format", "csv") \ .schema(schema) \ .load("abfss://my-bucket/csvData") \ .selectExpr("*", "_metadata as source_metadata") \ .writeStream \ .format("delta") \ .option("checkpointLocation", checkpointLocation) \ .start(targetTable)

topic Workflow file watch - capture filename trigger in Data Engineering

Workflow file watch - capture filename trigger

Re: Workflow file watch - capture filename trigger