How to stop a Streaming Job based on time of the week

[nolanlavender00]

I have an always-on job cluster triggering Spark Streaming jobs. I would like to stop this streaming job once a week to run table maintenance. I was looking to leverage the foreachBatch function to check a condition and stop the job accordingly.

[Kaniz]

Hi @Nolan Lavender​ ! My name is Kaniz, and I'm a technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers on the Forum have an answer to your questions first. Or else I will follow up shortly with a response.

[mathan_pillai · Accepted solution]

Hi @Nolan Lavender​ , For e.g. if want to stop streaming on Saturday, you could do something like the below. Below is just a pseudo code.

.foreachBatch{ (batchDF: DataFrame, batchId: Long) =>

if (date_format(current_timestamp(), "u") == 6) { //run commands to maintain the table }

Alternatively, You can calculate approximately how many micro batches are processed in a week and then you can periodically stop the streaming job. If your streaming is processing 100 microbatches in a week, then you can do something like below.

.foreachBatch{ (batchDF: DataFrame, batchId: Long) =>

if (batchId % 101 == 0) { //run commands to maintain the table }

[Kaniz]

Hi @Nolan Lavender​, How is it going?

Were you able to resolve your problem?