topic Re: How to stop continuous running streaming job over weekend in Data Engineering

How to stop continuous running streaming job over weekend

sanjay — Fri, 10 Feb 2023 17:43:46 GMT

I have an continuous running streaming Job, I would like to stop this over weekend and start again on Monday. Here is my streaming job code.

(spark.readStream.format("delta").load(input_path)

.writeStream

.option("checkpointLocation", input_checkpoint_path)

.trigger(processingTime="1 minute")

.foreachBatch(foreachBatchFunction)

.start()

)

Regards

Sanjay

Re: How to stop continuous running streaming job over weekend

Lakshay — Fri, 10 Feb 2023 19:17:59 GMT

Hi @Sanjay Jain , You can stop your job from the UI

Re: How to stop continuous running streaming job over weekend

MoJaMa — Sat, 11 Feb 2023 04:42:32 GMT

We're bringing "Continuous" jobs as a first-class citizen for exactly this scenario. So you will be able to "Pause" a job on Friday PM for example, and start again on Monday AM. Talk to your Databricks account team for access to the Private Preview.

Re: How to stop continuous running streaming job over weekend

sanjay — Sat, 11 Feb 2023 10:07:22 GMT

Thank you Lakshay. I am looking to stop automatically through some regex or programatically.

Its hard and chances of forgetting to stop every Friday and start again on Monday.

Re: How to stop continuous running streaming job over weekend

sanjay — Sat, 11 Feb 2023 10:07:58 GMT

Thank you Mathew. I will check Continuous job feature.

Re: How to stop continuous running streaming job over weekend

Anonymous — Mon, 13 Feb 2023 06:48:29 GMT

Hi @Sanjay Jain

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: How to stop continuous running streaming job over weekend

sanjay — Tue, 14 Feb 2023 12:09:11 GMT

Hi Vidula,

I could not find continuous option as suggested by Mathew. Can you help how/where to see this option.

Regards,

Sanjay

Re: How to stop continuous running streaming job over weekend

Data_Analytics1 — Thu, 16 Feb 2023 12:28:06 GMT

@Vidula Khanna I too couldn't find the continuous option or trigger option. @Mohan Mathews can you help us in getting the option?

Re: How to stop continuous running streaming job over weekend

NDK — Wed, 23 Aug 2023 17:41:28 GMT

@sanjay Any luck on that, I am also looking for the solution for the same issue