How to Schedule stop and start for a Continuous DLT pipeline

[surajitDE]

I have a use case where in there is a DLT pipeline in continuous mode, the requirement is to run this pipeline on scheduled basis i.e everyday it starts at 8am and stops at 8pm. I see a schedule option to start the job but don't see any option to stop it that is why I have to manually stop it everyday at 8pm else it continuously runs throughout. 

[Mike_Szklarczyk]

You can terminate any Job with REST API.

I recommend to use Python SDK jobs.cancel_all_runs() method

https://databricks-sdk-py.readthedocs.io/en/latest/workspace/jobs/jobs.html#databricks.sdk.service.j...

[surajitDE]

Consider this scenario: A Python SDK script is placed inside a notebook and runs on a jobs cluster. When my DLT pipeline is triggered, it uses the DLT cluster. This setup runs continuously for 8 hours(Problem is job cluster runs 8 hours just to run a simple SDK script), leading to unnecessary resource and time wastage

[Mike_Szklarczyk]

Do you can schedule this notebook as separate Job on dedicated job cluster that start everyday at 8pm.

I think that using Serverless cluster for this task will be the best way.