topic Re: How to make the job fail via code after handling exception in Data Engineering

How to make the job fail via code after handling exception

kjoth — Wed, 02 Feb 2022 07:38:45 GMT

Hi , We are capturing the exception if an error occurs using try except. But we want the job status to be failed once we got the exception. Whats the best way to do that.

We are using pyspark.

Re: How to make the job fail via code after handling exception

Hubert-Dudek — Wed, 02 Feb 2022 14:39:37 GMT

try:
 ...
except Exception as error:
  dbutils.notebook.exit(str(jobId) + ' - ERROR!!! - ' + repr(error))

Re: How to make the job fail via code after handling exception

kjoth — Fri, 04 Feb 2022 04:52:38 GMT

Would it work if we are not running notebook based jobs?

Re: How to make the job fail via code after handling exception

kjoth — Thu, 10 Feb 2022 09:29:47 GMT

Thanks for sharing the link. It doesn't help much for the case.

Re: How to make the job fail via code after handling exception

jose_gonzalez — Wed, 02 Mar 2022 01:57:32 GMT

Hi @karthick J ,

Please try the code below:

# Errors in workflows thrown a WorkflowException.

def run_with_retry(notebook, timeout, args = {}, max_retries = 3):

num_retries = 0

while True:

try:

return dbutils.notebook.run(notebook, timeout, args)

except Exception as e:

if num_retries > max_retries:

raise e

else:

print("Retrying error", e)

num_retries += 1

run_with_retry("LOCATION_OF_CALLEE_NOTEBOOK", 60, max_retries = 5)

For more info please check the docs https://docs.databricks.com/notebooks/notebook-workflows.html#handle-errors

Re: How to make the job fail via code after handling exception

Anonymous — Fri, 22 Apr 2022 14:28:01 GMT

Hi @karthick J

Hope you are well.

Just wanted to see if you were able to find an answer to your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

Cheers!

Re: How to make the job fail via code after handling exception

kjoth — Tue, 26 Apr 2022 11:06:37 GMT

No, didnt get the solution. Maybe there isnt one for this use case.

Re: How to make the job fail via code after handling exception

lcalca95 — Thu, 15 Dec 2022 15:43:06 GMT

Hi,

I've build a job on a python wheel and I've obtained the opposite behaviour (the job failed). I'm using a try/except like your. I've also tried to create a job from a simple notebook and now the job succeded!

Someone knows why the behaviour changes in relation to the format?

Re: How to make the job fail via code after handling exception

AkA — Thu, 04 May 2023 10:06:49 GMT

Instead of exiting the notebook which make the task/job success, Exception objects needs to be raised again from Exception block to fail the job.

try:

except Exception as err:

raise err

Re: How to make the job fail via code after handling exception

kumar_ravi — Wed, 04 Sep 2024 11:19:28 GMT

you can do some hack arround

dbutils = get_dbutils(spark)

tables_with_exceptions = []

for table_config in table_configs:

try:

process(spark, table_config)

except Exception as e:

exception_detail = f"Error processing table {table_config.table_name}: {e}"

tables_with_exceptions.append(exception_detail)

if tables_with_exceptions:

for exception in tables_with_exceptions:

print(exception)

raise RuntimeError(f"Job failed with exceptions: {tables_with_exceptions}")

else:

dbutils.notebook.exit("completed without exceptions")