topic Guidance Required: Scheduling a Biweekly Databricks Job in Data Engineering

Guidance Required: Scheduling a Biweekly Databricks Job

pawanswami — Tue, 06 Oct 2026 01:55:01 GMT

I’m reaching out to seek your guidance on one of our use cases where we need to schedule a biweekly Databricks job to run every alternate Wednesday.

I explored both the standard Databricks scheduling options and cron-based scheduling, but I was unable to configure the schedule to meet this requirement reliably.

Re: Guidance Required: Scheduling a Biweekly Databricks Job

Satyasai — Tue, 06 Oct 2026 04:19:15 GMT

Hi @pawanswami
The limitation you ran into is due to the underlying scheduling engine: Databricks uses Quartz Cron, which does not support "every 2 weeks" (n-week intervals) natively. A cron expression like 0 0 9 ? * WED will run every Wednesday, not every alternate Wednesday.

To handle alternate Wednesday scheduling reliably in Databricks Workflows, I use to work with below solution

Schedule Weekly + Early Exit Guard
You set the job schedule to run every Wednesday, but the first task in your workflow checks the ISO week number and stops execution if it is an "off" week.Step 1 Create a Check_Schedule TaskAdd a lightweight Python notebook/script task as the first step in your workflow Python import datetime

import sys

# Get current ISO week number (1 - 53)
current_week = datetime.datetime.now().isocalendar().week

# Option A: Even weeks vs. Odd weeks
is_run_week = (current_week % 2 == 0)

# Option B: Calculate from a fixed reference Wednesday (e.g., Jan 7, 2026)
# ref_date = datetime.datetime(2026, 1, 7)
# weeks_since_ref = (datetime.datetime.now() - ref_date).days // 7
# is_run_week = (weeks_since_ref % 2 == 0)

if not is_run_week:
print("Skipping execution: Current week is an off-week.")
# Exit task gracefully so downstream tasks do not run
dbutils.notebook.exit("SKIPPED_OFF_WEEK")

Re: Guidance Required: Scheduling a Biweekly Databricks Job

ivy125cordes — Tue, 06 Oct 2026 06:04:31 GMT

Standard cron expressions do not natively support alternating weeks, which is why Databricks' built-in scheduler cannot handle biweekly jobs directly. To resolve this, configure your job to run every Wednesday using cron (0 0 12 ? * WED) and insert a conditional check at the start of your script or notebook (such as evaluating whether the current week number is even or odd) to execute the workflow only on alternating weeks. You can verify this configuration was successful by checking your Databricks job run logs on consecutive Wednesdays to confirm it correctly alternates between executing and skipping.

Re: Guidance Required: Scheduling a Biweekly Databricks Job

pawanswami — Tue, 06 Oct 2026 06:21:07 GMT

@Satyasai that was really helpful. thanks

Re: Guidance Required: Scheduling a Biweekly Databricks Job

balajij8 — Tue, 06 Oct 2026 08:14:04 GMT

@pawanswami

You can set the schedule in the DAB for the task and start the pipeline on Wednesday. You can set the periodic trigger configured below (interval 2 unit WEEKS) to achieve the biweekly scheduling. You can try to un pause the job on a Wednesday and it will run every 2 weeks on Wednesdays from that point forward. You can also achieve it in a notebook to manage the scheduling.

resources: jobs: jobs_t: name: job_t trigger: pause_status: PAUSED periodic: interval: 2 unit: WEEKS tasks: - task_key: run_notebook notebook_task: notebook_path: /Users/biweekly_wednesday source: WORKSPACE

Re: Guidance Required: Scheduling a Biweekly Databricks Job

data_pulse — Tue, 06 Oct 2026 11:23:17 GMT

@pawanswami

I did a quick bundle validation of both options - periodic Trigger and Scheduled Trigger.

With a Periodic Trigger:

trigger: pause_status: UNPAUSED periodic: interval: 2 unit: WEEKS

Databricks created an “Every 2 weeks” schedule, but the exact first run time was chosen automatically when the schedule was created. In my test the UI showed a specific next-run date/time (as below) that I had not configured anywhere in the bundle

So this is a good fit when the requirement is simply “run every 2 weeks” and the exact weekday/time does not matter. It is interval-based rather than calendar-based, and there are no fields to specify Wednesday, 09:00, or a timezone.

I also tested how the periodic trigger behaves on bundle updates:

Redeploy with no trigger change: next-run timestamp stayed the same.
Changed 2 WEEKS to 3 WEEKS: Databricks recalculated the next run.
Changed back 3 WEEKS to 2 WEEKS: it recalculated again to a different timestamp than the original one.

So editing the periodic trigger can effectively re-anchor the schedule. That makes periodic suitable for “every N weeks,” but not ideal for a strict calendar requirement like “alternate Wednesdays at 09:00 Europe/London

With Cron Schedule: If the requirement is strictly “every other Wednesday at 09:00 Europe/London”, can use

schedule: quartz_cron_expression: "0 0 9 ? * WED" timezone_id: "Europe/London" pause_status: UNPAUSED

and then use a small first task referring a notebook to calculate whether this Wednesday is part of the 14-day cadence from a fixed reference Wednesday (like below):

from datetime import date reference = date(2026, 10, 7) today = date.today() should_run = ( today >= reference and (today - reference).days % 14 == 0 ) dbutils.jobs.taskValues.set( key="should_run", value=str(should_run).lower() )

Then configure the If/else condition to evaluate and run the task. For the bundle file, it looks mostly like this:

schedule: quartz_cron_expression: "0 0 9 ? * WED" timezone_id: "Europe/London" pause_status: UNPAUSED tasks: - task_key: check_biweekly_date spark_python_task: python_file: jobs/check_biweekly.py environment_key: default - task_key: is_biweekly_run depends_on: - task_key: check_biweekly_date condition_task: op: EQUAL_TO left: "{{tasks.check_biweekly_date.values.should_run}}" right: "true" - task_key: actual_processing depends_on: - task_key: is_biweekly_run outcome: "true" spark_python_task: python_file: jobs/process.py environment_key: default

This looks cleaner than embedding skip logic inside the processing notebook, because the Jobs UI clearly shows whether that Wednesday was a run or skip.

Re: Guidance Required: Scheduling a Biweekly Databricks Job

aayush_410 — Tue, 06 Oct 2026 13:08:58 GMT

Databricks schedules use Quartz cron, which has no way to say "every 14 days". Cron fields only match calendar positions (day-of-week, day-of-month), so you can't alternate weeks reliably. Patterns like 1,15,29 on days of the month drift and break at month boundaries.

Recommended: run weekly, skip alternate weeks with a gate task

Schedule the job every Wednesday, and add a first task that decides whether this is an "on" week. Downstream tasks run only if it says yes.

1. Schedule weekly (Quartz cron):

yaml
schedule:
quartz_cron_expression: "0 0 6 ? * WED *" # every Wednesday 06:00
timezone_id: "Asia/Kolkata" # set to your timezone
pause_status: UNPAUSED

2. Gate notebook (check_week): use an anchor date instead of ISO week numbers, since ISO week parity breaks at year boundaries (52/53-week years):

python
from datetime import date

ANCHOR = date(2026, 10, 7) # a Wednesday that SHOULD run; change as needed
weeks_since = (date.today() - ANCHOR).days // 7
should_run = (weeks_since % 2 == 0)

dbutils.jobs.taskValues.set(key="should_run", value=str(should_run).lower())

3. Wire it up with a condition task:

yaml
tasks:
- task_key: check_week
notebook_task:
notebook_path: jobs/base/CHECK_WEEK
source: GIT

- task_key: is_run_week
depends_on:
- task_key: check_week
condition_task:
op: EQUAL_TO
left: "{{tasks.check_week.values.should_run}}"
right: "true"

- task_key: main_processing
depends_on:
- task_key: is_run_week
outcome: "true"
notebook_task:
notebook_path: jobs/base/MAIN_PROCESSING
source: GIT

On "off" weeks, main_processing is shown as excluded and the run finishes successfully, with no failure alerts. The gate adds only a few seconds of compute, and you can run it on a small job cluster.

Alternative options

Databricks periodic trigger (trigger.periodic with interval: 2, unit: WEEKS). It does repeat every 2 weeks, but it isn't anchored to a weekday or time. It counts from when the job is created or saved, so you'd have to deploy it on the right Wednesday. Redeploying or editing the job can shift the cycle, so it's fragile in production.

Azure Data Factory (since you already use it). A Schedule trigger supports frequency: Week, interval: 2, weekDays: ["Wednesday"], with a startTime on the first run date. ADF then calls the Databricks job via the Jobs API or a Databricks activity. This is clean if ADF is already your orchestrator.

Two separate jobs or schedules: some teams use cron on specific dates, like 0 0 6 1,15,29 * ?, but this is not actually alternate Wednesdays and I don't recommend it.