Workflow file watch - capture filename trigger

[csmcpherson]

With respect to the file watch trigger in workflows, how can we capture what files and or path was identified as raising the trigger? 

I'd like to use this information to set parameters based upon the file name and the file path 

Thank you! 

https://docs.databricks.com/en/jobs/file-arrival-triggers.html

[szymon_dybczak · Accepted solution]

Hi @csmcpherson ,

This is currently not supported, but databricks team is working on that idea according to below thread:

Solved: File information is not passed to trigger job on f... - Databricks Community - 39266

As a workaround, if you use autoloader, you can use file _metadata column.

File metadata column - Azure Databricks | Microsoft Learn

spark.readStream \
  .format("cloudFiles") \
  .option("cloudFiles.format", "csv") \
  .schema(schema) \
  .load("abfss://my-bucket/csvData") \
  .selectExpr("*", "_metadata as source_metadata") \
  .writeStream \
  .format("delta") \
  .option("checkpointLocation", checkpointLocation) \
  .start(targetTable)