How to Schedule stop and start for a Continuous DLT pipeline
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-19-2024 02:24 AM
I have a use case where in there is a DLT pipeline in continuous mode, the requirement is to run this pipeline on scheduled basis i.e everyday it starts at 8am and stops at 8pm. I see a schedule option to start the job but don't see any option to stop it that is why I have to manually stop it everyday at 8pm else it continuously runs throughout.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-19-2024 05:21 AM
You can terminate any Job with REST API.
I recommend to use Python SDK jobs.cancel_all_runs() method
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-19-2024 05:59 AM
Consider this scenario: A Python SDK script is placed inside a notebook and runs on a jobs cluster. When my DLT pipeline is triggered, it uses the DLT cluster. This setup runs continuously for 8 hours(Problem is job cluster runs 8 hours just to run a simple SDK script), leading to unnecessary resource and time wastage
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-19-2024 07:58 AM
Do you can schedule this notebook as separate Job on dedicated job cluster that start everyday at 8pm.
I think that using Serverless cluster for this task will be the best way.
