Guidance Required: Scheduling a Biweekly Databricks Job

[pawanswami]

I’m reaching out to seek your guidance on one of our use cases where we need to schedule a biweekly Databricks job to run every alternate Wednesday.

I explored both the standard Databricks scheduling options and cron-based scheduling, but I was unable to configure the schedule to meet this requirement reliably.

[Satyasai]

Hi @pawanswami 
The limitation you ran into is due to the underlying scheduling engine: Databricks uses Quartz Cron, which does not support "every 2 weeks" (n-week intervals) natively. A cron expression like 0 0 9 ? * WED will run every Wednesday, not every alternate Wednesday.

To handle alternate Wednesday scheduling reliably in Databricks Workflows, I use to work with below solution

Schedule Weekly + Early Exit Guard
You set the job schedule to run every Wednesday, but the first task in your workflow checks the ISO week number and stops execution if it is an "off" week.Step 1 Create a Check_Schedule TaskAdd a lightweight Python notebook/script task as the first step in your workflow Python import datetime

import sys

# Get current ISO week number (1 - 53)
current_week = datetime.datetime.now().isocalendar().week

# Option A: Even weeks vs. Odd weeks
is_run_week = (current_week % 2 == 0)

# Option B: Calculate from a fixed reference Wednesday (e.g., Jan 7, 2026)
# ref_date = datetime.datetime(2026, 1, 7)
# weeks_since_ref = (datetime.datetime.now() - ref_date).days // 7
# is_run_week = (weeks_since_ref % 2 == 0)

if not is_run_week:
print("Skipping execution: Current week is an off-week.")
# Exit task gracefully so downstream tasks do not run
dbutils.notebook.exit("SKIPPED_OFF_WEEK")

[pawanswami]

@Satyasai that was really helpful. thanks

[ivy125cordes]

Standard cron expressions do not natively support alternating weeks, which is why Databricks' built-in scheduler cannot handle biweekly jobs directly. To resolve this, configure your job to run every Wednesday using cron (0 0 12 ? * WED) and insert a conditional check at the start of your script or notebook (such as evaluating whether the current week number is even or odd) to execute the workflow only on alternating weeks. You can verify this configuration was successful by checking your Databricks job run logs on consecutive Wednesdays to confirm it correctly alternates between executing and skipping.

[balajij8]

@pawanswami 

You can set the schedule in the DAB for the task and start the pipeline on Wednesday. You can set the periodic trigger configured below (interval 2 unit WEEKS) to achieve the biweekly scheduling. You can try to un pause the job on a Wednesday and it will run every 2 weeks on Wednesdays from that point forward. You can also achieve it in a notebook to manage the scheduling.

resources:
  jobs:
    jobs_t:
      name: job_t
      trigger:
        pause_status: PAUSED
        periodic:
          interval: 2
          unit: WEEKS
      tasks:
        - task_key: run_notebook
          notebook_task:
            notebook_path: /Users/biweekly_wednesday
            source: WORKSPACE

[data_pulse]

@pawanswami 

I did a quick bundle validation of both options - periodic Trigger and Scheduled Trigger.

With a Periodic Trigger: 

trigger:
  pause_status: UNPAUSED
  periodic:
    interval: 2
    unit: WEEKS

Databricks created an “Every 2 weeks” schedule, but the exact first run time was chosen automatically when the schedule was created. In my test the UI showed a specific next-run date/time (as below) that I had not configured anywhere in the bundle

So this is a good fit when the requirement is simply “run every 2 weeks” and the exact weekday/time does not matter. It is interval-based rather than calendar-based, and there are no fields to specify Wednesday, 09:00, or a timezone.

I also tested how the periodic trigger behaves on bundle updates:

• Redeploy with no trigger change: next-run timestamp stayed the same.
• Changed 2 WEEKS to 3 WEEKS: Databricks recalculated the next run.
• Changed back 3 WEEKS to 2 WEEKS: it recalculated again to a different timestamp than the original one.

So editing the periodic trigger can effectively re-anchor the schedule. That makes periodic suitable for “every N weeks,” but not ideal for a strict calendar requirement like “alternate Wednesdays at 09:00 Europe/London

With Cron Schedule: If the requirement is strictly “every other Wednesday at 09:00 Europe/London”, can use

schedule:
  quartz_cron_expression: "0 0 9 ? * WED"
  timezone_id: "Europe/London"
  pause_status: UNPAUSED

and then use a small first task referring a notebook to calculate whether this Wednesday is part of the 14-day cadence from a fixed reference Wednesday (like below):

from datetime import date

reference = date(2026, 10, 7)
today = date.today()

should_run = (
    today >= reference
    and (today - reference).days % 14 == 0
)

dbutils.jobs.taskValues.set(
    key="should_run",
    value=str(should_run).lower()
)

Then configure the If/else condition to evaluate and run the task. For the bundle file, it looks mostly like this:

schedule:
  quartz_cron_expression: "0 0 9 ? * WED"
  timezone_id: "Europe/London"
  pause_status: UNPAUSED

tasks:
  - task_key: check_biweekly_date
    spark_python_task:
      python_file: jobs/check_biweekly.py
    environment_key: default

  - task_key: is_biweekly_run
    depends_on:
      - task_key: check_biweekly_date
    condition_task:
      op: EQUAL_TO
      left: "{{tasks.check_biweekly_date.values.should_run}}"
      right: "true"

  - task_key: actual_processing
    depends_on:
      - task_key: is_biweekly_run
        outcome: "true"
    spark_python_task:
      python_file: jobs/process.py
    environment_key: default

This looks cleaner than embedding skip logic inside the processing notebook, because the Jobs UI clearly shows whether that Wednesday was a run or skip.

[aayush_410]

Databricks schedules use Quartz cron, which has no way to say "every 14 days". Cron fields only match calendar positions (day-of-week, day-of-month), so you can't alternate weeks reliably. Patterns like 1,15,29 on days of the month drift and break at month boundaries.

Recommended: run weekly, skip alternate weeks with a gate task

Schedule the job every Wednesday, and add a first task that decides whether this is an "on" week. Downstream tasks run only if it says yes.

1. Schedule weekly (Quartz cron):

yaml
schedule:
quartz_cron_expression: "0 0 6 ? * WED *" # every Wednesday 06:00
timezone_id: "Asia/Kolkata" # set to your timezone
pause_status: UNPAUSED

2. Gate notebook (check_week): use an anchor date instead of ISO week numbers, since ISO week parity breaks at year boundaries (52/53-week years):

python
from datetime import date

ANCHOR = date(2026, 10, 7) # a Wednesday that SHOULD run; change as needed
weeks_since = (date.today() - ANCHOR).days // 7
should_run = (weeks_since % 2 == 0)

dbutils.jobs.taskValues.set(key="should_run", value=str(should_run).lower())

3. Wire it up with a condition task:

yaml
tasks:
- task_key: check_week
notebook_task:
notebook_path: jobs/base/CHECK_WEEK
source: GIT

- task_key: is_run_week
depends_on:
- task_key: check_week
condition_task:
op: EQUAL_TO
left: "{{tasks.check_week.values.should_run}}"
right: "true"

- task_key: main_processing
depends_on:
- task_key: is_run_week
outcome: "true"
notebook_task:
notebook_path: jobs/base/MAIN_PROCESSING
source: GIT

On "off" weeks, main_processing is shown as excluded and the run finishes successfully, with no failure alerts. The gate adds only a few seconds of compute, and you can run it on a small job cluster.

Alternative options

Databricks periodic trigger (trigger.periodic with interval: 2, unit: WEEKS). It does repeat every 2 weeks, but it isn't anchored to a weekday or time. It counts from when the job is created or saved, so you'd have to deploy it on the right Wednesday. Redeploying or editing the job can shift the cycle, so it's fragile in production.

Azure Data Factory (since you already use it). A Schedule trigger supports frequency: Week, interval: 2, weekDays: ["Wednesday"], with a startTime on the first run date. ADF then calls the Databricks job via the Jobs API or a Databricks activity. This is clean if ADF is already your orchestrator.

Two separate jobs or schedules: some teams use cron on specific dates, like 0 0 6 1,15,29 * ?, but this is not actually alternate Wednesdays and I don't recommend it.