How to stop continuous running streaming job over weekend

[sanjay]

I have an continuous running streaming Job, I would like to stop this over weekend and start again on Monday. Here is my streaming job code.

(spark.readStream.format("delta").load(input_path)

.writeStream

.option("checkpointLocation", input_checkpoint_path)

.trigger(processingTime="1 minute")

.foreachBatch(foreachBatchFunction)

.start()

)

Regards

Sanjay

[Lakshay]

Hi @Sanjay Jain​ , You can stop your job from the UI

[sanjay]

Thank you Lakshay. I am looking to stop automatically through some regex or programatically.

Its hard and chances of forgetting to stop every Friday and start again on Monday.

[MoJaMa]

We're bringing "Continuous" jobs as a first-class citizen for exactly this scenario. So you will be able to "Pause" a job on Friday PM for example, and start again on Monday AM. Talk to your Databricks account team for access to the Private Preview.

[sanjay]

Thank you Mathew. I will check Continuous job feature.

[Anonymous]

Hi @Sanjay Jain​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

[sanjay]

Hi Vidula,

I could not find continuous option as suggested by Mathew. Can you help how/where to see this option.

Regards,

Sanjay

[Data_Analytics1]

@Vidula Khanna​  I too couldn't find the continuous option or trigger option. @Mohan Mathews​  can you help us in getting the option?

[NDK]

@sanjay Any luck on that, I am also looking for the solution for the same issue