topic Run continuous job for a period of time in Data Engineering

Run continuous job for a period of time

theanhdo — Mon, 11 Nov 2024 00:16:57 GMT

Hi there,

I have a job where the Trigger type is configured as Continuous. I want to only run the Continuous job for a period of time per day, e.g. 8AM - 5PM. I understand that we can achieve it by manually starting and cancelling the job on the UI, or by programmatically starting and cancelling the job using these APIs
https://<databricks-instance>/api/2.1/jobs/run-now
https://<databricks-instance>/api/2.1/jobs/runs/cancel

However, I would like to ask if there is any job setting, e.g. using cron syntax, to achieve this?

Re: Run continuous job for a period of time

MuthuLakshmi — Mon, 11 Nov 2024 12:24:11 GMT

@theanhdo

To schedule a job to run at 8 AM every day, you should use the Scheduled trigger type rather than the Continuous trigger type. The Continuous trigger type is designed to keep a job running continuously, which is not suitable for running a job at a specific time each day.

Here’s how you can schedule a job to run at 8 AM every day using the Scheduled trigger type:

Navigate to Workflows:

In the Databricks workspace, go to the sidebar and click on Workflows.

Select the Job:

Click the job name in the Name column on the Jobs tab.

Add a Trigger:

In the Job details panel, click Add trigger.

Configure the Trigger:

In the Trigger type dropdown, select Scheduled.
In the Schedule type dropdown, select Advanced.

Set the Schedule:

Use the following cron expression to schedule the job to run at 8 AM every day:

0 8 * * *

Optionally, select the Show Cron Syntax checkbox to display and edit the schedule using Quartz Cron Syntax.

Save the Configuration:

Click Save to apply the schedule.

This configuration will ensure that your job runs at 8 AM every day.

To stop the job, you have to use rest API

You will need to create a separate job or task that stops the main job at 5 PM. This can be done using the Databricks REST API to cancel the job run.

Create a new job that uses the REST API to cancel the main job run.

Add a trigger to this new job with the following cron expression to run at 5 PM every da

Re: Run continuous job for a period of time

theanhdo — Mon, 11 Nov 2024 21:22:37 GMT

Hi @MuthuLakshmi , thank you for your answer. However, your answer doesn't help with my question. Let me rephrase my question.

In short, my question is how to configure a Continuous job to run for a period of time, e.g. from 8AM to 5PM every day, and automatically stop in other time of the day?

In details, I have a job that is running Continuously from 8AM to 5PM every day, and in other time of the day I want to stop it. The job is configured with the Trigger set as Continuous, however there is no option to configure the running period. I understand that we can achieve it by manually starting and cancelling the job on the UI, or by programmatically starting and cancelling the job using these APIs
https://<databricks-instance>/api/2.1/jobs/run-now
https://<databricks-instance>/api/2.1/jobs/runs/cancel

However, I would like to ask if there is any job setting, e.g. using cron syntax, to achieve this?

Re: Run continuous job for a period of time

eslaats — Wed, 11 Dec 2024 16:44:39 GMT

There doesn't seem to be a proper way to do this currently.

We ended up running the job a couple of times in order to figure out some upper bound for run time, and just using that in the cron. Some jobs now run every 5 minutes during office hours, which is close enough for our usecase.

This does cause issues with Skipped runs when compute is slow to spin up, so make sure you adjust any notifications accordingly.

Alternatively, one could apply a Continuous schedule to the job, then toggle the schedule state for that job to ACTIVE at the start and to PAUSED at the end of the day using the Databricks API. We added two jobs in Databricks that call the API to toggle this state. Do test this thoroughly, or you'll have some costs waiting for you by Monday morning 🙂

This all feels very hacky for functionality that feels like it should be supported by default.

Re: Run continuous job for a period of time

Leon_543 — Tue, 12 May 2026 17:44:33 GMT

I completely agree with eslaats it feels incredibly hacky that Databricks doesn't support business-hour windows for Continuous triggers out of the box!

We ran into this exact same billing nightmare. Leaving a cluster running continuously overnight just to process a trickle of streaming data was destroying our budget. We ended up adopting the second approach mentioned above: using a lightweight external automation (like an AWS EventBridge cron triggering a Lambda) to hit the Databricks API. It toggles the job's schedule state to ACTIVE in the morning and PAUSED in the evening.

The only tricky part was aligning those API triggers with our actual shifting business schedules. We had to strictly map out our team's core hours to ensure the continuous job was active exactly when the data analysts were actively querying the downstream tables, without leaving it hanging open and burning compute during holidays or partial shifts.

It takes a bit of initial setup outside of the Databricks UI, but using the REST API to pause the schedule state is definitely the most robust and cost-effective workaround until they natively support time-boxing!

Re: Run continuous job for a period of time

KrisJohannesen — Tue, 12 May 2026 17:53:20 GMT

The "not-so-pretty-but-it-works" solution I have come across is exactly what you are hinting at yourself.

Create the Continuous job - have it be paused
Create a secondary "start job"-job - which is basically just that API call in a notebook or python file, on a Cron schedule
Create a third "stop job"-job. The flipside of the above - also scheduled
Cry a little that it took 3 jobs to achieve what one job should do on its own - but be happy it does what it is supposed to do.