- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-13-2022 11:36 AM
Accepted Solutions

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-13-2022 04:17 PM
You can use the CLI https://docs.databricks.com/dev-tools/cli/index.html to schedule a job and you can also schedule a cluster to terminate through API calls. There are also integrations with tools such as airflow and azure data factory.

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-13-2022 04:17 PM
You can use the CLI https://docs.databricks.com/dev-tools/cli/index.html to schedule a job and you can also schedule a cluster to terminate through API calls. There are also integrations with tools such as airflow and azure data factory.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-02-2022 10:19 AM
You could set up schedule to start 6am and timeout seconds to 39 600 that is 11 hours. With max retries to 1. There is a downside to it - if your stream fails in the middle of day - it will run for 11 hours regardless when it stops.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-14-2022 01:50 PM
If you are looking for a graceful stop (Not to stop exactly at 5 but stop after the micro-batch that was in progress at 5 o clock instead of abruptly stopping the stream), you can try the following. The downside is if the micro batch duration is high, stream stop will be delayed.
import java.time.LocalTime
val queryStopListner = new StreamingQueryListener() {
override def onQueryStarted(queryStarted: StreamingQueryListener.QueryStartedEvent): Unit = {
}
override def onQueryTerminated(queryTerminated: StreamingQueryListener.QueryTerminatedEvent): Unit = {
}
override def onQueryProgress(queryProgress: StreamingQueryListener.QueryProgressEvent): Unit = {
val id = queryProgress.progress.id
if(LocalTime.now().isAfter(LocalTime.parse("17:00:00"))){
val currentStreamingQuery = spark.streams.get(id)
currentStreamingQuery.stop
}
}
}
//Add this query listner to the session
spark.streams.addListener(queryStopListner)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-22-2023 02:00 PM
Might anybody knows what will happen if I set task timeout in Workflows for the Streaming job?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-23-2023 02:45 AM
If you are streaming to delta, not much, the micro batch will fail and in next time the stream will pick up from last successful write (due to ACID). I don't know about other formats, what happens if the stream is aborted in mid micro batch.

