Works for any event-driven workload: IoT alerts, e-commerce flash sales, financial market close processing.

Goal

In this project, I needed to start Databricks jobs on an irregular basis, driven entirely by timestamps stored in PostgreSQL rather than by a fixed schedule.

The concrete use case was processing football match data right after the final whistle. Because matches have irregular kick-off times and variable durations, it was not possible to define a simple, fixed schedule for job runs. Instead, the requirement was to trigger a job at `match_start_time + 105 minutes`, which corresponds to the earliest possible end time of a match.

Challenge

On a previous project, I had solved a similar problem using Apache Airflow. There, I could rely on a timetable schedule: an external source with a collection of timestamps that defines when a DAG (Airflow job) should be triggered. Airflow would automatically poll that source and trigger the DAG at the appropriate times.

When I moved to Databricks, I could not find a direct equivalent of this timetable-style scheduling. Out of the box, Databricks jobs can be triggered in the following ways:

- Using a CRON schedule
- On file arrival
- On table update (not yet available at the time of this implementation)
- Via REST API

None of these options provided the same "pull timestamps from an external list and schedule dynamically" behavior that I previously had with Airflow timetables.

Idea: A Self-Rescheduling Orchestrator Job

To bridge this gap, I designed an orchestrator job responsible for triggering ETL jobs at the specific timestamps stored in PostgreSQL.

The high-level approach was:

• When the orchestrator job runs, it queries PostgreSQL for match data based on the job execution timestamp
• It selects all matches that start around "now" plus a time buffer (for example, all matches that start at the current time plus some offset)
• For each such game, the orchestrator triggers the appropriate ETL jobs
• After triggering those ETL jobs, the orchestrator reschedules **itself** to run at the next relevant timestamp taken from PostgreSQL

What happens if there are no upcoming matches in PostgreSQL?
There is one more component running at fixed schedules: a job that fetches new match datetimes and populates PostgreSQL. It is also responsible for checking whether the orchestrator is scheduled; if not, it schedules it for the earliest upcoming match.

Why not run-now? A natural alternative might be using the REST API run-now endpoint in a polling loop from an external scheduler. However, cron updates are preferable here because they enable precise, native scheduling without external timers or continuous polling.

• Second-level precision: Cron updates let Databricks handle exact timing using its managed scheduler—no drift from external cron jobs or polling intervals
• Zero idle compute: Unlike polling every X minutes, the job sleeps until the exact timestamp, minimizing costs
• Fault tolerance: Databricks retries failed schedules automatically; external polling would need custom retry logic
• Simplicity: One job self-manages its lifecycle vs. maintaining separate poller + trigger infrastructure.

This keeps everything within Databricks while achieving Airflow timetable-like behavior.

Why not use newer Databricks features like table triggers or Spark Declarative Pipelines?
By the time of implementation, table update triggers weren't available. Now they enable event-driven job execution when Delta/Iceberg tables are updated (merge/delete), with dynamic parameters like commit timestamp. However, they require mirroring PostgreSQL data into Delta tables first, adding sync overhead and latency—unsuitable for direct timestamp-driven scheduling from an external relational database.

Spark Declarative Pipelines (part of Lakeflow) excel at no-code ETL orchestration with automated lineage and retries, but their scheduling still relies on CRON, file arrival, or table triggers—not dynamic querying of external PostgreSQL timestamps to compute precise execution windows.

This self-rescheduling is the key part of the solution. It turns the job into a precise, event-driven loop that reprograms its next execution time according to the latest information in PostgreSQL—without requiring continuous running and polling of the database, which would incur unnecessary costs and wasted compute time, unlike a static CRON expression defined once.

Implementation: Updating the Job Schedule via REST API

Rescheduling the job is done through the Databricks REST API, using the `/api/2.2/jobs/update` endpoint and providing a new schedule in the `new_settings` object.

Converting Python datetime to Quartz Cron format
Databricks uses Quartz Cron expressions for schedules, which follow the format: `[second] [minute] [hour] [day of month] [month] [day of week] [year]`. I created a helper function to convert a Python `datetime.datetime` object to this format:

def datetime_to_quartz_cron(dt: datetime) -> str:
    """Convert datetime to Quartz cron expression.
    Quartz cron format: [second] [minute] [hour] [day of month] [month] [day of week] [year]
    http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html

    Parameters
    ----------
    dt : datetime.datetime
        Datetime to convert

    Returns
    -------
    str
        Quartz cron expression
    """
    return f"{dt.second} {dt.minute} {dt.hour} {dt.day} {dt.month} ? {dt.year}"

The core Python snippet that performs the reschedule looks like this:

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.jobs import CronSchedule, JobSettings

new_cron_schedule = CronSchedule(
                quartz_cron_expression=new_schedule_date_quartz_cron_format,
                timezone_id="UTC",
            )
new_settings = JobSettings(schedule=new_cron_schedule)

workspace_client = WorkspaceClient(**config)
workspace_client.jobs.update(job_id=job_id, new_settings=new_settings)

In this pattern:

• The orchestrator computes `new_schedule_date_quartz_cron_format` (using the helper function above) based on the next batch of matches stored in PostgreSQL
• It then constructs a new `CronSchedule` and wraps it into `JobSettings`
• Finally, it calls `jobs.update` for its own `job_id`, effectively changing its next trigger time to align with the next match window

This provides a flexible, data-driven schedule: as soon as match data changes in PostgreSQL, the orchestrator will adapt its future runs accordingly, without manual intervention or static CRON maintenance.