topic Re: File_arrival trigger in Workflow in Data Engineering

File_arrival trigger in Workflow

NavyaSinghvi — Mon, 01 Jul 2024 11:40:50 GMT

I am using "job.trigger.file_arrival.location" in job parameters to get triggered file location . But I am getting error "job.trigger.file_arrival.location is not allowed". How can I get triggered file location in workflow ?

Re: File_arrival trigger in Workflow

brockb — Mon, 01 Jul 2024 16:27:15 GMT

Hi @NavyaSinghvi ,

Can you please confirm the Requirements and Limitations listed in this doc:

https://docs.databricks.com/en/workflows/jobs/file-arrival-triggers.html#requirements

If each of those are met, perhaps you could share a screenshot of your setup with the error message?

Thank you.

Re: File_arrival trigger in Workflow

NavyaSinghvi — Tue, 02 Jul 2024 12:59:05 GMT

Hi @brockb ,

All requirements from documents are met. I am able to trigger workflow based on file arrival. But can not use job.trigger.file_arrival.location in job parameters.

Re: File_arrival trigger in Workflow

brockb — Wed, 03 Jul 2024 03:33:01 GMT

Hi @NavyaSinghvi ,

Thank you for clarifying that. I believe I was able to replicate the issue you're describing and was able to circumvent it by adding `job.trigger.file_arrival.location` as a task-level parameter, and not as a job-level parameter.

Can you please give that a try and let me know how it goes?

Thank you.

Re: File_arrival trigger in Workflow

NavyaSinghvi — Wed, 03 Jul 2024 10:27:00 GMT

Hi @brockb ,

At task level it is working fine. Thanks.

Re: File_arrival trigger in Workflow

raghu2 — Fri, 17 Jan 2025 03:22:00 GMT

@brockb , I added parameter at both the job and task level. Value is not replaced by actual name of the file that triggered the job. Can you please let me know what I am missing? Thanks,

job_parameters=[JobParameter(default='{{job.trigger.file_arrival.location}}', name='loc', value=None)]

base_parameters={'loc1': '{{job.trigger.file_arrival.location}}', 'location': '{{job.trigger.file_arrival.location}}', 'loc2': '{{job.parameters.loc}}', 'job.trigger.file_arrival.location': '{{job.trigger.file_arrival.location}}', 'loc3': '{{job.trigger.file_arrival.location}}'}

Re: File_arrival trigger in Workflow

raghu2 — Fri, 17 Jan 2025 20:06:30 GMT

The parameters are passed as widgets to the job. After defining the parameters in the job definition, With following code I was able to access the data associated with the parameter:

widget_names = ["loc1", "loc2", "loc3"] # Add all expected parameter names here

print("Parameters passed to the notebook:")

for name in widget_names:

try:

value = dbutils.widgets.get(name)

print(f"{name}: {value}")

except Exception as e:

print(f"{name}: Not defined (Error: {str(e)})")