Shutdown Cluster in script if there is any failure

[sahesh1320]

I am working on incremental load from sql server to Delta lake tables stored in ADLS gen2. DUring the script i need to qrite a logic to

• Shut down the DB cluster on failure (there needs to be logging added to ensure that shutdown happens promptly to prevent guardrail violations). 

What is the optimal way to achieve this?

[Wojciech_BUK]

If you run your notebook via workflow and error happen and there are no retires on job, then job cluster will be terminated immidietly after failure.

You can add python block of try catch and if error occurs , you catch the error and log somewhere before job will be terminated