Shutdown Cluster in script if there is any failure
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-22-2023 09:16 AM
I am working on incremental load from sql server to Delta lake tables stored in ADLS gen2. DUring the script i need to qrite a logic to
- Shut down the DB cluster on failure (there needs to be logging added to ensure that shutdown happens promptly to prevent guardrail violations).
What is the optimal way to achieve this?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-22-2023 11:00 AM
If you run your notebook via workflow and error happen and there are no retires on job, then job cluster will be terminated immidietly after failure.
You can add python block of try catch and if error occurs , you catch the error and log somewhere before job will be terminated
