topic Re: Guidance Required: Scheduling a Biweekly Databricks Job in Data Engineering

Guidance Required: Scheduling a Biweekly Databricks Job

pawanswami — Tue, 06 Oct 2026 01:55:01 GMT

I’m reaching out to seek your guidance on one of our use cases where we need to schedule a biweekly Databricks job to run every alternate Wednesday.

I explored both the standard Databricks scheduling options and cron-based scheduling, but I was unable to configure the schedule to meet this requirement reliably.

Re: Guidance Required: Scheduling a Biweekly Databricks Job

Satyasai — Tue, 06 Oct 2026 04:19:15 GMT

Hi @pawanswami
The limitation you ran into is due to the underlying scheduling engine: Databricks uses Quartz Cron, which does not support "every 2 weeks" (n-week intervals) natively. A cron expression like 0 0 9 ? * WED will run every Wednesday, not every alternate Wednesday.

To handle alternate Wednesday scheduling reliably in Databricks Workflows, I use to work with below solution

Schedule Weekly + Early Exit Guard
You set the job schedule to run every Wednesday, but the first task in your workflow checks the ISO week number and stops execution if it is an "off" week.Step 1 Create a Check_Schedule TaskAdd a lightweight Python notebook/script task as the first step in your workflow Python import datetime

import sys

# Get current ISO week number (1 - 53)
current_week = datetime.datetime.now().isocalendar().week

# Option A: Even weeks vs. Odd weeks
is_run_week = (current_week % 2 == 0)

# Option B: Calculate from a fixed reference Wednesday (e.g., Jan 7, 2026)
# ref_date = datetime.datetime(2026, 1, 7)
# weeks_since_ref = (datetime.datetime.now() - ref_date).days // 7
# is_run_week = (weeks_since_ref % 2 == 0)

if not is_run_week:
print("Skipping execution: Current week is an off-week.")
# Exit task gracefully so downstream tasks do not run
dbutils.notebook.exit("SKIPPED_OFF_WEEK")

Re: Guidance Required: Scheduling a Biweekly Databricks Job

ivy125cordes — Tue, 06 Oct 2026 06:04:31 GMT

Standard cron expressions do not natively support alternating weeks, which is why Databricks' built-in scheduler cannot handle biweekly jobs directly. To resolve this, configure your job to run every Wednesday using cron (0 0 12 ? * WED) and insert a conditional check at the start of your script or notebook (such as evaluating whether the current week number is even or odd) to execute the workflow only on alternating weeks. You can verify this configuration was successful by checking your Databricks job run logs on consecutive Wednesdays to confirm it correctly alternates between executing and skipping.

Re: Guidance Required: Scheduling a Biweekly Databricks Job

pawanswami — Tue, 06 Oct 2026 06:21:07 GMT

@Satyasai that was really helpful. thanks

Re: Guidance Required: Scheduling a Biweekly Databricks Job

balajij8 — Tue, 06 Oct 2026 08:14:04 GMT

@pawanswami

You can set the schedule in the DAB for the task and start the pipeline on Wednesday. You can set the periodic trigger configured below (interval 2 unit WEEKS) to achieve the biweekly scheduling. You can try to un pause the job on a Wednesday and it will run every 2 weeks on Wednesdays from that point forward. You can also achieve it in a notebook to manage the scheduling.

resources: jobs: jobs_t: name: job_t trigger: pause_status: PAUSED periodic: interval: 2 unit: WEEKS tasks: - task_key: run_notebook notebook_task: notebook_path: /Users/biweekly_wednesday source: WORKSPACE

Re: Guidance Required: Scheduling a Biweekly Databricks Job

data_pulse — Tue, 06 Oct 2026 11:23:17 GMT

@pawanswami

I did a quick bundle validation of both options - periodic Trigger and Scheduled Trigger.

With a Periodic Trigger:

trigger: pause_status: UNPAUSED periodic: interval: 2 unit: WEEKS

Databricks created an “Every 2 weeks” schedule, but the exact first run time was chosen automatically when the schedule was created. In my test the UI showed a specific next-run date/time (as below) that I had not configured anywhere in the bundle

So this is a good fit when the requirement is simply “run every 2 weeks” and the exact weekday/time does not matter. It is interval-based rather than calendar-based, and there are no fields to specify Wednesday, 09:00, or a timezone.

I also tested how the periodic trigger behaves on bundle updates:

Redeploy with no trigger change: next-run timestamp stayed the same.
Changed 2 WEEKS to 3 WEEKS: Databricks recalculated the next run.
Changed back 3 WEEKS to 2 WEEKS: it recalculated again to a different timestamp than the original one.

So editing the periodic trigger can effectively re-anchor the schedule. That makes periodic suitable for “every N weeks,” but not ideal for a strict calendar requirement like “alternate Wednesdays at 09:00 Europe/London

With Cron Schedule: If the requirement is strictly “every other Wednesday at 09:00 Europe/London”, can use

schedule: quartz_cron_expression: "0 0 9 ? * WED" timezone_id: "Europe/London" pause_status: UNPAUSED

and then use a small first task referring a notebook to calculate whether this Wednesday is part of the 14-day cadence from a fixed reference Wednesday (like below):

from datetime import date reference = date(2026, 10, 7) today = date.today() should_run = ( today >= reference and (today - reference).days % 14 == 0 ) dbutils.jobs.taskValues.set( key="should_run", value=str(should_run).lower() )

Then configure the If/else condition to evaluate and run the task. For the bundle file, it looks mostly like this:

schedule: quartz_cron_expression: "0 0 9 ? * WED" timezone_id: "Europe/London" pause_status: UNPAUSED tasks: - task_key: check_biweekly_date spark_python_task: python_file: jobs/check_biweekly.py environment_key: default - task_key: is_biweekly_run depends_on: - task_key: check_biweekly_date condition_task: op: EQUAL_TO left: "{{tasks.check_biweekly_date.values.should_run}}" right: "true" - task_key: actual_processing depends_on: - task_key: is_biweekly_run outcome: "true" spark_python_task: python_file: jobs/process.py environment_key: default

This looks cleaner than embedding skip logic inside the processing notebook, because the Jobs UI clearly shows whether that Wednesday was a run or skip.

Re: Guidance Required: Scheduling a Biweekly Databricks Job

aayush_410 — Tue, 06 Oct 2026 13:08:58 GMT

Databricks schedules use Quartz cron, which has no way to say "every 14 days". Cron fields only match calendar positions (day-of-week, day-of-month), so you can't alternate weeks reliably. Patterns like 1,15,29 on days of the month drift and break at month boundaries.

Recommended: run weekly, skip alternate weeks with a gate task

Schedule the job every Wednesday, and add a first task that decides whether this is an "on" week. Downstream tasks run only if it says yes.

1. Schedule weekly (Quartz cron):

yaml
schedule:
quartz_cron_expression: "0 0 6 ? * WED *" # every Wednesday 06:00
timezone_id: "Asia/Kolkata" # set to your timezone
pause_status: UNPAUSED

2. Gate notebook (check_week): use an anchor date instead of ISO week numbers, since ISO week parity breaks at year boundaries (52/53-week years):

python
from datetime import date

ANCHOR = date(2026, 10, 7) # a Wednesday that SHOULD run; change as needed
weeks_since = (date.today() - ANCHOR).days // 7
should_run = (weeks_since % 2 == 0)

dbutils.jobs.taskValues.set(key="should_run", value=str(should_run).lower())

3. Wire it up with a condition task:

yaml
tasks:
- task_key: check_week
notebook_task:
notebook_path: jobs/base/CHECK_WEEK
source: GIT

- task_key: is_run_week
depends_on:
- task_key: check_week
condition_task:
op: EQUAL_TO
left: "{{tasks.check_week.values.should_run}}"
right: "true"

- task_key: main_processing
depends_on:
- task_key: is_run_week
outcome: "true"
notebook_task:
notebook_path: jobs/base/MAIN_PROCESSING
source: GIT

On "off" weeks, main_processing is shown as excluded and the run finishes successfully, with no failure alerts. The gate adds only a few seconds of compute, and you can run it on a small job cluster.

Alternative options

Databricks periodic trigger (trigger.periodic with interval: 2, unit: WEEKS). It does repeat every 2 weeks, but it isn't anchored to a weekday or time. It counts from when the job is created or saved, so you'd have to deploy it on the right Wednesday. Redeploying or editing the job can shift the cycle, so it's fragile in production.

Azure Data Factory (since you already use it). A Schedule trigger supports frequency: Week, interval: 2, weekDays: ["Wednesday"], with a startTime on the first run date. ADF then calls the Databricks job via the Jobs API or a Databricks activity. This is clean if ADF is already your orchestrator.

Two separate jobs or schedules: some teams use cron on specific dates, like 0 0 6 1,15,29 * ?, but this is not actually alternate Wednesdays and I don't recommend it.

Re: Guidance Required: Scheduling a Biweekly Databricks Job

Louis_Frolio — Wed, 07 Oct 2026 15:11:10 GMT

Greetings @pawanswami, I did some digging and here is what I found.

Satyasai and ivy125cordes have the root cause right: Databricks schedules run on Quartz cron, and Quartz has no "every N weeks" concept. The question that decides your fix is whether you need a two-week interval, or a calendar-anchored schedule like "every other Wednesday at 9:00 local time." Those are different requirements with different solutions.

Option 1: Simple schedule, every 2 weeks

This is what balajij8 showed with interval: 2 and unit: WEEKS. Least code, and a true two-week interval. The catch, straight from the docs: you can't set the time of the first run on a Simple schedule. The scheduler picks it when you save, and the interval is anchored to that moment. So you'd create it on a Wednesday, but the time of day isn't yours to choose, and any later edit or redeploy re-anchors it. If "roughly every other Wednesday" is good enough, take this and test it after any trigger change.

Option 2: Weekly cron plus a gate task

If the business expects a specific Wednesday at a specific time, schedule weekly with an Advanced schedule and let the first task decide whether this is an "on" week. That's the pattern Satyasai and ivy125cordes described, with two adjustments.

First, don't key off ISO week parity (week % 2). Some ISO years have 53 weeks, and 2026 is one. December 30, 2026 is week 53 and January 6, 2027 is week 1, both odd, so an odd/even check would either run two Wednesdays in a row or go three weeks dark at year end. Count from a fixed anchor Wednesday instead (Satyasai's Option B). That's deterministic across year boundaries.

Second, dbutils.notebook.exit() ends the task as Succeeded, so downstream tasks with the default "All succeeded" run-if condition still fire. Publish a task value from the gate and branch on it with an If/else condition task. Off-week tasks then show as Excluded in the run history, and the skip decision is visible in the DAG instead of buried in a notebook.

Schedule (shown in bundle yaml; the UI equivalent is Advanced with cron syntax):

schedule:
  quartz_cron_expression: "0 0 9 ? * WED"
  timezone_id: "Europe/London"   # replace with yours
  pause_status: UNPAUSED

Gate task (Python notebook), first in the graph:

from datetime import date, datetime
from zoneinfo import ZoneInfo

TZ = ZoneInfo("Europe/London")   # must match timezone_id above
ANCHOR = date(2026, 10, 7)       # any Wednesday that should be an "on" week
today = datetime.now(TZ).date()  # plain date.today() can be wrong near midnight UTC

should_run = (today - ANCHOR).days // 7 % 2 == 0
dbutils.jobs.taskValues.set(key="should_run", value=str(should_run).lower())

If/else condition task that depends on the gate, with condition {{tasks.gate.values.should_run}} == true. Hang the real work off the (true) branch and leave (false) empty.

Tip: pass ANCHOR in as a job parameter so a cadence shift is a config change, not a code change.

At the end of the day, if the exact time doesn't matter, balajij8's periodic schedule is the answer. If it does, the weekly cron plus an anchored If/else gate gives you a predictable alternate-Wednesday job that survives year boundaries.

References:

Run jobs on a schedule (Simple vs. Advanced, first-run time note): https://docs.databricks.com/aws/en/jobs/scheduled
If/else condition task: https://docs.databricks.com/aws/en/jobs/tasks/if-else
Task values: https://docs.databricks.com/aws/en/jobs/task-values
Run tasks conditionally (run-if rules): https://docs.databricks.com/workflows/jobs/conditional-tasks.html
Periodic trigger example in a bundle: https://docs.databricks.com/dev-tools/bundles/resource-examples.html
Quartz cron syntax: https://www.quartz-scheduler.org/documentation/quartz-2.5.x/tutorials/tutorial-lesson-06.html

Regards, Louis.