Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-05-2024 03:35 PM
With respect to the file watch trigger in workflows, how can we capture what files and or path was identified as raising the trigger?
I'd like to use this information to set parameters based upon the file name and the file path
Thank you!
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-06-2024 02:50 AM
Hi @csmcpherson ,
This is currently not supported, but databricks team is working on that idea according to below thread:
Solved: File information is not passed to trigger job on f... - Databricks Community - 39266
As a workaround, if you use autoloader, you can use file _metadata column.
File metadata column - Azure Databricks | Microsoft Learn
spark.readStream \
.format("cloudFiles") \
.option("cloudFiles.format", "csv") \
.schema(schema) \
.load("abfss://my-bucket/csvData") \
.selectExpr("*", "_metadata as source_metadata") \
.writeStream \
.format("delta") \
.option("checkpointLocation", checkpointLocation) \
.start(targetTable)