notebook returns succeeded even if there is a failure in any of the command

[moulitharan]

I'm trying to run a notebook that has 14 commands from adf notebook activity, I'm writing the transformed data to a delta table in the last command. I'm handling the last command in try except block to handle errors, and raising error on exception.

when I run the notebook from adf, the job run status shows succeeded even if any of the command fails. I need the notebook status to be returned as failure even if any of the command fails. But if there is a failure in any command the following commands are skipped. Sometimes even on raising exception in try except block, the status of the notebook is still displayed as succeeded.

Is there a way to get the actual status of the notebook job run.

[Kaniz_Fatma]

Hi @moulitharan, When running a Databricks Notebook using the Databricks Notebook Activity in Azure Data Factory (ADF), the job status is typically reported as “succeeded” even if individual commands within the notebook fail. However, there are ways to handle this and get more accurate status information.

1. Monitor Databricks Job Status:

   • After running the notebook, you can log in to your Azure Databricks workspace, navigate to the Clusters section, and find the job associated with your notebook.
   • The job status will be displayed as either “pending execution,” “running,” or “terminated.”
   • Click on the job name to view further details, including any errors or exceptions encountered during execution.
   • This approach allows you to validate the parameters passed and the output of the Python notebook¹.

2. Custom Logic in ADF:

   • To handle the scenario where you want the notebook status to be returned as “failure” even if any command fails, you can implement custom logic in your ADF pipeline.
   • One approach is to use an additional activity (e.g., a copy data activity) after the notebook activity.
   • In this subsequent activity, you can check the status of the notebook execution (e.g., using an expression like @equals(activity('NotebookActivity').output.executionDetails.status,'Succeeded')).
   • If the status is “Succeeded,” you can raise an error or take other appropriate actions to indicate failure.
   • This way, you can ensure that the overall pipeline status reflects the success or failure of the ent...².

3. Pipeline Run History:

   • In the Azure Data Factory portal, go to the Monitor tab and select Pipeline runs.
   • Locate the pipeline that contains the notebook activity.
   • Open the pipeline run history to see details about the notebook run, including input and output for each cell.
   • This information can help you troubleshoot any issues and understand the overall execution status³.

If you encounter any specific issues or need further assistance, feel free to ask! 😊