I want to run a streaming job from morning 6a.m to evening 5p.m how can I schedule this window in databricks. Or how can u stop my stream at 5pm?

[Anonymous · Accepted solution]

You can use the CLI https://docs.databricks.com/dev-tools/cli/index.html to schedule a job and you can also schedule a cluster to terminate through API calls. There are also integrations with tools such as airflow and azure data factory.

[merca]

You could set up schedule to start 6am and timeout seconds to 39 600 that is 11 hours. With max retries to 1. There is a downside to it - if your stream fails in the middle of day - it will run for 11 hours regardless when it stops.

[Sandeep]

If you are looking for a graceful stop (Not to stop exactly at 5 but stop after the micro-batch that was in progress at 5 o clock instead of abruptly stopping the stream), you can try the following. The downside is if the micro batch duration is high, stream stop will be delayed.

import java.time.LocalTime
val queryStopListner = new StreamingQueryListener() {
    override def onQueryStarted(queryStarted: StreamingQueryListener.QueryStartedEvent): Unit = {
      
    }
    override def onQueryTerminated(queryTerminated: StreamingQueryListener.QueryTerminatedEvent): Unit = {
      
    }
    override def onQueryProgress(queryProgress: StreamingQueryListener.QueryProgressEvent): Unit = {
      
      val id = queryProgress.progress.id
      if(LocalTime.now().isAfter(LocalTime.parse("17:00:00"))){
        val currentStreamingQuery = spark.streams.get(id)  
        currentStreamingQuery.stop
      }
    }
}
 
//Add this query listner to the session
spark.streams.addListener(queryStopListner)

[DimaP]

Might anybody knows what will happen if I set task timeout in Workflows for the Streaming job?

[merca]

If you are streaming to delta, not much, the micro batch will fail and in next time the stream will pick up from last successful write (due to ACID). I don't know about other formats, what happens if the stream is aborted in mid micro batch.