Databricks execution failed with error state: InternalError, error message: failed to update run

[dataengineer17]

I am receiving this error Databricks execution failed with error state: InternalError, error message: failed to update run GlobalRunId(xx,RunId(yy))

This is appears as an error message in azure data factory when I use it to schedule a databricks notebook.  When I click the notebook run, I see this message as well (Please refer to screenshots)

Appreciate any help on this as this is appearing as a false error on ADF as the dababricks notebook still run smoothly.

[Ajay-Pandey]

Hi @dataengineer17 ,

It looks like databricks internal error that you are getting, please create a support ticket with databricks .

[Kaniz_Fatma]

Hi @dataengineer17 , Double-check your code and configuration to ensure no issues specific to your code. Also, please verify that all required dependencies are available and correctly set up in the Databricks environment. Then, Retry the execution to see if the error persists. It's possible that the issue was temporary and resolved itself upon retry.

[dataengineer17]

Thanks for your responses.

 @Kaniz_Fatma it is not code or configuration issue.  The job runs successfully for other runs, that run with issue was an exceptional case.

 @ajay yes it sure looks like databricks internal issue. Was wondering if anyone has faced similiar issue

[jose_gonzalez]

Are you able to check the driver's logs for more details?

[dataengineer17]

@jose_gonzalez  yup I checked all driver logs (stdout, stderr, log4j) but nothing unusual

[saipujari_spark]

@dataengineer17 

It could be coming from the internal jobs service, If the issue persists I would recommend creating a support ticket.