File_arrival trigger in Workflow

[NavyaSinghvi]

I am using  "job.trigger.file_arrival.location" in job parameters to get triggered file location . But I am getting error "job.trigger.file_arrival.location is not allowed". How can I get triggered file location in workflow ?

[brockb]

Hi @NavyaSinghvi ,

Can you please confirm the Requirements and Limitations listed in this doc:

https://docs.databricks.com/en/workflows/jobs/file-arrival-triggers.html#requirements

If each of those are met, perhaps you could share a screenshot of your setup with the error message?

Thank you.

[NavyaSinghvi]

Hi @brockb ,

All requirements from documents are met. I am able to trigger workflow based on file arrival. But can not use job.trigger.file_arrival.location in job parameters.

[brockb · Accepted solution]

Hi @NavyaSinghvi ,

Thank you for clarifying that. I believe I was able to replicate the issue you're describing and was able to circumvent it by adding `job.trigger.file_arrival.location` as a task-level parameter, and not as a job-level parameter.

Can you please give that a try and let me know how it goes?

Thank you.

[NavyaSinghvi]

Hi @brockb ,

At task level it is working fine. Thanks. 

[raghu2]

@brockb , I added parameter at both the job and task level. Value is not replaced by actual name of the file that triggered the job. Can you please let me know what I am missing? Thanks,

job_parameters=[JobParameter(default='{{job.trigger.file_arrival.location}}', name='loc', value=None)]

base_parameters={'loc1': '{{job.trigger.file_arrival.location}}', 'location': '{{job.trigger.file_arrival.location}}', 'loc2': '{{job.parameters.loc}}', 'job.trigger.file_arrival.location': '{{job.trigger.file_arrival.location}}', 'loc3': '{{job.trigger.file_arrival.location}}'}

[raghu2]

The parameters are passed as widgets to the job. After defining the parameters in the job definition, With following code I was able to access the data associated with the parameter:

widget_names = ["loc1", "loc2", "loc3"]  # Add all expected parameter names here

print("Parameters passed to the notebook:")

for name in widget_names:

    try:

        value = dbutils.widgets.get(name)

        print(f"{name}: {value}")

    except Exception as e:

        print(f"{name}: Not defined (Error: {str(e)})")