cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How to stop continuous running streaming job over weekend

sanjay
Valued Contributor II

I have an continuous running streaming Job, I would like to stop this over weekend and start again on Monday. Here is my streaming job code.

(spark.readStream.format("delta").load(input_path)

.writeStream

.option("checkpointLocation", input_checkpoint_path)

.trigger(processingTime="1 minute")

.foreachBatch(foreachBatchFunction)

.start()

)

Regards

Sanjay

8 REPLIES 8

Lakshay
Databricks Employee
Databricks Employee

Hi @Sanjay Jainโ€‹ , You can stop your job from the UI

sanjay
Valued Contributor II

Thank you Lakshay. I am looking to stop automatically through some regex or programatically.

Its hard and chances of forgetting to stop every Friday and start again on Monday.

MoJaMa
Databricks Employee
Databricks Employee

We're bringing "Continuous" jobs as a first-class citizen for exactly this scenario. So you will be able to "Pause" a job on Friday PM for example, and start again on Monday AM. Talk to your Databricks account team for access to the Private Preview.

Continuous Job

sanjay
Valued Contributor II

Thank you Mathew. I will check Continuous job feature.

Anonymous
Not applicable

Hi @Sanjay Jainโ€‹ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

sanjay
Valued Contributor II

Hi Vidula,

I could not find continuous option as suggested by Mathew. Can you help how/where to see this option.

Regards,

Sanjay

@Vidula Khannaโ€‹  I too couldn't find the continuous option or trigger option. @Mohan Mathewsโ€‹  can you help us in getting the option?

NDK
New Contributor II

@sanjay Any luck on that, I am also looking for the solution for the same issue

 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group