https://community.databricks.com/t5/data-engineering/shutdown-cluster-in-script-if-there-is-any-failure/m-p/55673#M30403 <P>I am working on incremental load from sql server to Delta lake tables stored in ADLS gen2. DUring the script i need to qrite a logic to</P><UL><LI>Shut down the DB cluster on failure (there needs to be logging added to ensure that shutdown happens promptly to prevent guardrail violations).&nbsp;</LI></UL><P>What is the optimal way to achieve this?</P> Fri, 22 Dec 2023 17:16:35 GMT sahesh1320 2023-12-22T17:16:35Z https://community.databricks.com/t5/data-engineering/shutdown-cluster-in-script-if-there-is-any-failure/m-p/55673#M30403 <P>I am working on incremental load from sql server to Delta lake tables stored in ADLS gen2. DUring the script i need to qrite a logic to</P><UL><LI>Shut down the DB cluster on failure (there needs to be logging added to ensure that shutdown happens promptly to prevent guardrail violations).&nbsp;</LI></UL><P>What is the optimal way to achieve this?</P> Fri, 22 Dec 2023 17:16:35 GMT https://community.databricks.com/t5/data-engineering/shutdown-cluster-in-script-if-there-is-any-failure/m-p/55673#M30403 sahesh1320 2023-12-22T17:16:35Z https://community.databricks.com/t5/data-engineering/shutdown-cluster-in-script-if-there-is-any-failure/m-p/55677#M30406 <P>If you run your notebook via workflow and error happen and there are no retires on job, then job cluster will be terminated immidietly after failure.</P><P>You can add python block of try catch and if error occurs , you catch the error and log somewhere before job will be terminated&nbsp;</P> Fri, 22 Dec 2023 19:00:28 GMT https://community.databricks.com/t5/data-engineering/shutdown-cluster-in-script-if-there-is-any-failure/m-p/55677#M30406 Wojciech_BUK 2023-12-22T19:00:28Z