File_arrival trigger in Workflow

NavyaSinghvi · ‎07-01-2024

I am using "job.trigger.file_arrival.location" in job parameters to get triggered file location . But I am getting error "job.trigger.file_arrival.location is not allowed". How can I get triggered file location in workflow ?

brockb · ‎07-01-2024

Hi @NavyaSinghvi ,

Can you please confirm the Requirements and Limitations listed in this doc:

https://docs.databricks.com/en/workflows/jobs/file-arrival-triggers.html#requirements

If each of those are met, perhaps you could share a screenshot of your setup with the error message?

Thank you.

NavyaSinghvi · ‎07-02-2024

Hi @brockb ,

All requirements from documents are met. I am able to trigger workflow based on file arrival. But can not use job.trigger.file_arrival.location in job parameters.

brockb · ‎07-02-2024

Hi @NavyaSinghvi ,

Thank you for clarifying that. I believe I was able to replicate the issue you're describing and was able to circumvent it by adding `job.trigger.file_arrival.location` as a task-level parameter, and not as a job-level parameter.

Can you please give that a try and let me know how it goes?

Thank you.

View solution in original post

NavyaSinghvi · ‎07-03-2024

Hi @brockb ,

At task level it is working fine. Thanks.

raghu2 · ‎01-16-2025

@brockb , I added parameter at both the job and task level. Value is not replaced by actual name of the file that triggered the job. Can you please let me know what I am missing? Thanks,

job_parameters=[JobParameter(default='{{job.trigger.file_arrival.location}}', name='loc', value=None)]

base_parameters={'loc1': '{{job.trigger.file_arrival.location}}', 'location': '{{job.trigger.file_arrival.location}}', 'loc2': '{{job.parameters.loc}}', 'job.trigger.file_arrival.location': '{{job.trigger.file_arrival.location}}', 'loc3': '{{job.trigger.file_arrival.location}}'}

raghu2 · ‎01-17-2025

The parameters are passed as widgets to the job. After defining the parameters in the job definition, With following code I was able to access the data associated with the parameter:

widget_names = ["loc1", "loc2", "loc3"] # Add all expected parameter names here

print("Parameters passed to the notebook:")

for name in widget_names:

try:

value = dbutils.widgets.get(name)

print(f"{name}: {value}")

except Exception as e:

print(f"{name}: Not defined (Error: {str(e)})")