topic Re: Multiple schedules in workflow with different parameters in Data Engineering

Multiple schedules in workflow with different parameters

Alby091 — Tue, 07 Jan 2025 21:02:31 GMT

I have a notebook that takes a file from the landing, processes it and saves a delta table.
This notebook contains a parameter (time_prm) that allows you to do this option for the different versions of files that arrive every day.

Specifically, for each file, I will have 3 daily versions and the nomenclature is the following file_202401070900, file_202401071200, file_202401071500.

I have a workflow that launches this notebook but I need to schedule it 3 times a day and for each build of the job, I will need to take the relative file.

For example, in the 0900 schedule I will need to take the file version file_202401070900, so I will insert 0900 as the time_prm parameter. For the file file_202401071200 I will insert 1200 etc.

Can I create multiple schedules of the same workflow and set a different time parameter for each schedule?

Do you have any other ideas? It seems strange to me that I have to create 3 different workflows.

Thanks for your help!

Re: Multiple schedules in workflow with different parameters

Walter_C — Wed, 08 Jan 2025 12:10:53 GMT

Right now jobs support only 1 Schedule process per job, as you have mentioned you will need to create a different job for each schedule you require, you can use the clone capability to facilitate the process.

Re: Multiple schedules in workflow with different parameters

ImranA — Thu, 23 Oct 2025 11:35:59 GMT

You can do multiple schedules with Cron expression. If you are using a Cron expression in Databricks asset bundle YAML, but the limitation is you can't have one running at 0 past the hour and another at 25 past.
i.e:
quartz_cron_expression: 0 45 9,23 * * ?

you can only do 8:30, 22: 30 and vice versa so the minutes will stay the same.