cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to stop continuous running streaming job over weekend

sanjay
Valued Contributor II

I have an continuous running streaming Job, I would like to stop this over weekend and start again on Monday. Here is my streaming job code.

(spark.readStream.format("delta").load(input_path)

.writeStream

.option("checkpointLocation", input_checkpoint_path)

.trigger(processingTime="1 minute")

.foreachBatch(foreachBatchFunction)

.start()

)

Regards

Sanjay

8 REPLIES 8

Lakshay
Esteemed Contributor
Esteemed Contributor

Hi @Sanjay Jain​ , You can stop your job from the UI

sanjay
Valued Contributor II

Thank you Lakshay. I am looking to stop automatically through some regex or programatically.

Its hard and chances of forgetting to stop every Friday and start again on Monday.

MoJaMa
Valued Contributor II
Valued Contributor II

We're bringing "Continuous" jobs as a first-class citizen for exactly this scenario. So you will be able to "Pause" a job on Friday PM for example, and start again on Monday AM. Talk to your Databricks account team for access to the Private Preview.

Continuous Job

sanjay
Valued Contributor II

Thank you Mathew. I will check Continuous job feature.

Anonymous
Not applicable

Hi @Sanjay Jain​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

sanjay
Valued Contributor II

Hi Vidula,

I could not find continuous option as suggested by Mathew. Can you help how/where to see this option.

Regards,

Sanjay

@Vidula Khanna​  I too couldn't find the continuous option or trigger option. @Mohan Mathews​  can you help us in getting the option?

NDK
New Contributor II

@sanjay Any luck on that, I am also looking for the solution for the same issue

 

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!