topic Re: How can I set the status of a databricks job as skipped via python? in Data Engineering

How can I set the status of a databricks job as skipped via python?

Michael_Papadop — Fri, 07 Apr 2023 13:35:05 GMT

I have a basic 2 task job.

The 1st notebook (task) checks whether the source file has changes and if so then refreshes a corresponding materialized view. In case we have no changes then I use dbutils.jobs.taskValues.set(key = "skip_job", value = 1) & dbutils.notebook.exit("SKIP").
The 2nd notebook (task) checks the passed variable from the 1st notebook with skip_job = dbutils.jobs.taskValues.get(taskKey = "01_prepare_fms_rmf_data", key = "skip_job", default = 0, debugValue = 0) and skips execution accordingly if (skip_job == 1): dbutils.notebook.exit("SKIP")

This seems to work fine for my case but I have I minor issue. I want to receive email notification upon Success or failure but not upon skip. The job Status is Success even when I use the above skip approach.

Is there a solution to this? Maybe there is some other command I can use to set the job status as skipped?

Re: How can I set the status of a databricks job as skipped via python?

karthik_p — Fri, 07 Apr 2023 18:09:54 GMT

@Michael Papadopoulos please select below option in job notification settings , that should handle u r case

Re: How can I set the status of a databricks job as skipped via python?

Michael_Papadop — Fri, 07 Apr 2023 20:09:26 GMT

Sorry, I forgot to mention that I had already tried that flag with no success. The job still sends notification even upon skip of both tasks.

Also the Statuses of both tasks and whole job is Succeeded.

Re: How can I set the status of a databricks job as skipped via python?

karthik_p — Fri, 07 Apr 2023 20:32:12 GMT

@Michael Papadopoulos usually that should not be the case i think, as for task level we have 3 level notifications ( success, failure,start), where as whole job level skip option is available to discard notification . will see if some one from community faced or respond