topic Re: stop autoloader with continuous trigger programatically in Get Started Discussions

stop autoloader with continuous trigger programatically

sanjay — Thu, 01 Feb 2024 18:31:36 GMT

Hi,

I am running autoloader with continuous trigger. How can I stop this trigger during some specific time, only if no data pending and current batch process is complete. How to check how many records pending in queue and current state.

Regards,

Sanjay

Re: stop autoloader with continuous trigger programatically

melbourne — Fri, 02 Feb 2024 13:33:34 GMT

You can switch to 'Triggered' pipeline in this case.

Next, create a job in workflow and attach a trigger of type 'file arrival' to it. Next add the notebook and cluster to the job. If you're not using DLT, then set cluster timeout minutes to 0, so that your cluster shuts down immediately after it's inactive.

Now, whenever a file will arrive in your landing location, this trigger will go off and will start the cluster which will then run the notebook until it finishes the task.

Re: stop autoloader with continuous trigger programatically

sanjay — Sat, 03 Feb 2024 14:27:30 GMT

Thank you melbourne. I can not switch to triggered pipeline for now. Is it possible to stop/pause using workspace client or Job Rest API?

Thanks,

Sanjay

Re: stop autoloader with continuous trigger programatically

RamonaMraz — Fri, 09 Feb 2024 06:37:28 GMT

Hello, I am new here, Can I ask a question?