topic Re: How to Schedule stop and start for a Continuous DLT pipeline in Data Engineering

How to Schedule stop and start for a Continuous DLT pipeline

surajitDE — Tue, 19 Nov 2024 10:24:18 GMT

I have a use case where in there is a DLT pipeline in continuous mode, the requirement is to run this pipeline on scheduled basis i.e everyday it starts at 8am and stops at 8pm. I see a schedule option to start the job but don't see any option to stop it that is why I have to manually stop it everyday at 8pm else it continuously runs throughout.

Re: How to Schedule stop and start for a Continuous DLT pipeline

Mike_Szklarczyk — Tue, 19 Nov 2024 13:21:53 GMT

You can terminate any Job with REST API.

I recommend to use Python SDK jobs.cancel_all_runs() method

https://databricks-sdk-py.readthedocs.io/en/latest/workspace/jobs/jobs.html#databricks.sdk.service.jobs.JobsExt.cancel_all_runs

Re: How to Schedule stop and start for a Continuous DLT pipeline

surajitDE — Tue, 19 Nov 2024 13:59:09 GMT

Consider this scenario: A Python SDK script is placed inside a notebook and runs on a jobs cluster. When my DLT pipeline is triggered, it uses the DLT cluster. This setup runs continuously for 8 hours(Problem is job cluster runs 8 hours just to run a simple SDK script), leading to unnecessary resource and time wastage

Re: How to Schedule stop and start for a Continuous DLT pipeline

Mike_Szklarczyk — Tue, 19 Nov 2024 15:58:06 GMT

Do you can schedule this notebook as separate Job on dedicated job cluster that start everyday at 8pm.

I think that using Serverless cluster for this task will be the best way.