cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How can I set the status of a databricks job as skipped via python?

Michael_Papadop
New Contributor II

I have a basic 2 task job.

  • The 1st notebook (task) checks whether the source file has changes and if so then refreshes a corresponding materialized view. In case we have no changes then I use dbutils.jobs.taskValues.set(key = "skip_job", value = 1) & dbutils.notebook.exit("SKIP").
  • The 2nd notebook (task) checks the passed variable from the 1st notebook with skip_job = dbutils.jobs.taskValues.get(taskKey = "01_prepare_fms_rmf_data", key = "skip_job", default = 0, debugValue = 0) and skips execution accordingly if (skip_job == 1): dbutils.notebook.exit("SKIP")

This seems to work fine for my case but I have I minor issue. I want to receive email notification upon Success or failure but not upon skip. The job Status is Success even when I use the above skip approach.

Is there a solution to this? Maybe there is some other command I can use to set the job status as skipped?

3 REPLIES 3

karthik_p
Esteemed Contributor

@Michael Papadopoulos​ please select below option in job notification settings , that should handle u r caseimage

Sorry, I forgot to mention that I had already tried that flag with no success. The job still sends notification even upon skip of both tasks.

Also the Statuses of both tasks and whole job is Succeeded.

image 

imageimage 

image

karthik_p
Esteemed Contributor

@Michael Papadopoulos​ usually that should not be the case i think, as for task level we have 3 level notifications ( success, failure,start), where as whole job level skip option is available to discard notification . will see if some one from community faced or respond

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group