topic Re: I want to run a streaming job from morning 6a.m to evening 5p.m how can I schedule this window in databricks. Or how can u stop my stream at 5pm? in Data Engineering

I want to run a streaming job from morning 6a.m to evening 5p.m how can I schedule this window in databricks. Or how can u stop my stream at 5pm?

Bhawna_bedi — Sun, 13 Mar 2022 18:36:10 GMT

Re: I want to run a streaming job from morning 6a.m to evening 5p.m how can I schedule this window in databricks. Or how can u stop my stream at 5pm?

Anonymous — Sun, 13 Mar 2022 23:17:32 GMT

You can use the CLI https://docs.databricks.com/dev-tools/cli/index.html to schedule a job and you can also schedule a cluster to terminate through API calls. There are also integrations with tools such as airflow and azure data factory.

Re: I want to run a streaming job from morning 6a.m to evening 5p.m how can I schedule this window in databricks. Or how can u stop my stream at 5pm?

merca — Sat, 02 Apr 2022 17:19:38 GMT

You could set up schedule to start 6am and timeout seconds to 39 600 that is 11 hours. With max retries to 1. There is a downside to it - if your stream fails in the middle of day - it will run for 11 hours regardless when it stops.

Re: I want to run a streaming job from morning 6a.m to evening 5p.m how can I schedule this window in databricks. Or how can u stop my stream at 5pm?

Sandeep — Thu, 14 Jul 2022 20:50:51 GMT

If you are looking for a graceful stop (Not to stop exactly at 5 but stop after the micro-batch that was in progress at 5 o clock instead of abruptly stopping the stream), you can try the following. The downside is if the micro batch duration is high, stream stop will be delayed.

import java.time.LocalTime
val queryStopListner = new StreamingQueryListener() {
    override def onQueryStarted(queryStarted: StreamingQueryListener.QueryStartedEvent): Unit = {
      
    }
    override def onQueryTerminated(queryTerminated: StreamingQueryListener.QueryTerminatedEvent): Unit = {
      
    }
    override def onQueryProgress(queryProgress: StreamingQueryListener.QueryProgressEvent): Unit = {
      
      val id = queryProgress.progress.id
      if(LocalTime.now().isAfter(LocalTime.parse("17:00:00"))){
        val currentStreamingQuery = spark.streams.get(id)  
        currentStreamingQuery.stop
      }
    }
}
 
//Add this query listner to the session
spark.streams.addListener(queryStopListner)

Re: I want to run a streaming job from morning 6a.m to evening 5p.m how can I schedule this window in databricks. Or how can u stop my stream at 5pm?

DimaP — Mon, 22 May 2023 21:00:41 GMT

Might anybody knows what will happen if I set task timeout in Workflows for the Streaming job?

Re: I want to run a streaming job from morning 6a.m to evening 5p.m how can I schedule this window in databricks. Or how can u stop my stream at 5pm?

merca — Tue, 23 May 2023 09:45:51 GMT

If you are streaming to delta, not much, the micro batch will fail and in next time the stream will pick up from last successful write (due to ACID). I don't know about other formats, what happens if the stream is aborted in mid micro batch.