@PiotrPustola -- The self-rescheduling orchestrator pattern is a really elegant solution for event-driven workloads that depend on externally managed timestamps. A few thoughts and additions that might help you and others who land on this article:

ADDITIONAL TIPS FOR THE SELF-RESCHEDULING PATTERN

1. Pause status awareness: When you update the schedule via the Jobs API, make sure to explicitly set pause_status to "UNPAUSED" in the CronSchedule object. If a job was paused for any reason (manual intervention, maintenance, etc.), just updating the quartz_cron_expression alone will not unpause it. Here is an example:

new_cron_schedule = CronSchedule(
quartz_cron_expression=new_schedule_date_quartz_cron_format,
timezone_id="UTC",
pause_status=PauseStatus.UNPAUSED
)

2. Idempotency and race conditions: If the orchestrator fails partway through (after triggering ETL jobs but before rescheduling itself), you could end up in a state where it never runs again. A good safety net is a separate lightweight "watchdog" job on a fixed daily CRON that checks whether the orchestrator has a valid future schedule, and reschedules it if not. It sounds like your match-fetching job already handles this, which is a solid design.

3. Scheduler latency: The Databricks documentation notes that the job scheduler is not designed for sub-second precision and may experience delays of up to several minutes due to infrastructure conditions. For your use case (triggering 105 minutes after kickoff) this is fine, but anyone adapting this pattern for tighter timing windows should be aware of this.

4. Using the Reset vs Update endpoint: You correctly use the Update endpoint (POST /api/2.2/jobs/update), which only modifies the fields you specify. Be careful not to accidentally use the Reset endpoint (POST /api/2.2/jobs/reset), which overwrites the entire job configuration and could wipe out task definitions, cluster settings, and other configuration if you only pass in the schedule.

TABLE UPDATE TRIGGERS AS A COMPLEMENT

You mentioned that table update triggers were not available at the time of your implementation. Now that they are GA, they could work well as a complementary pattern for some variations of this problem. Specifically, if you already have a job writing match schedule data into a Delta table (rather than only PostgreSQL), you could use a table update trigger to kick off downstream ETL whenever new match records land.

Table update triggers support dynamic parameters like commit timestamps and updated table lists, which can be useful for filtering logic in downstream tasks. Key limitations to be aware of: maximum 10 tables per trigger, and for best performance you should enable file events on the external storage location.

That said, for your exact use case -- scheduling at a computed future time (kickoff + 105 min) rather than reacting immediately to a data change -- the self-rescheduling approach is still the right tool. Table triggers fire on data arrival, not at a computed offset from a timestamp value in the data.

Documentation: https://docs.databricks.com/aws/en/jobs/trigger-table-update

WEBHOOK AND API-BASED ALTERNATIVES

For anyone reading this who does have an external system that can push events (rather than store timestamps for polling), the Jobs API run-now endpoint (POST /api/2.2/jobs/run-now) is also worth considering. You can trigger a job run immediately and pass parameters:

https://docs.databricks.com/api/workspace/jobs/runnow

This works well when paired with cloud-native event systems (AWS EventBridge, Azure Event Grid, GCP Pub/Sub) that can call webhooks or Lambda functions to trigger jobs via the API. But as you noted, this requires external infrastructure to manage the timing, which is exactly what your pattern avoids.

RELEVANT DOCUMENTATION

- Scheduled triggers (CRON): https://docs.databricks.com/aws/en/jobs/scheduled
- Table update triggers: https://docs.databricks.com/aws/en/jobs/trigger-table-update
- File arrival triggers: https://docs.databricks.com/aws/en/jobs/file-arrival-triggers
- Jobs API - Update endpoint: https://docs.databricks.com/api/workspace/jobs/update
- Jobs API - Run Now endpoint: https://docs.databricks.com/api/workspace/jobs/runnow
- Databricks SDK for Python: https://docs.databricks.com/aws/en/dev-tools/sdk-python
- Quartz Cron syntax reference: http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html

Thanks for sharing this -- the pattern generalizes nicely beyond sports data to any domain with externally defined, irregular event timestamps (IoT maintenance windows, financial market closes, logistics ETAs, etc.).

* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.

* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.