topic Databricks execution failed with error state: InternalError, error message: failed to update run in Data Engineering

Databricks execution failed with error state: InternalError, error message: failed to update run

dataengineer17 — Sun, 16 Jul 2023 16:01:08 GMT

I am receiving this error Databricks execution failed with error state: InternalError, error message: failed to update run GlobalRunId(xx,RunId(yy))

This is appears as an error message in azure data factory when I use it to schedule a databricks notebook. When I click the notebook run, I see this message as well (Please refer to screenshots)

Appreciate any help on this as this is appearing as a false error on ADF as the dababricks notebook still run smoothly.

Re: Databricks execution failed with error state: InternalError, error message: failed to update run

Ajay-Pandey — Mon, 17 Jul 2023 06:09:46 GMT

Hi @dataengineer17 ,

It looks like databricks internal error that you are getting, please create a support ticket with databricks .

Re: Databricks execution failed with error state: InternalError, error message: failed to update run

dataengineer17 — Mon, 17 Jul 2023 10:55:44 GMT

Thanks for your responses.

@Retired_mod it is not code or configuration issue. The job runs successfully for other runs, that run with issue was an exceptional case.

@ajay yes it sure looks like databricks internal issue. Was wondering if anyone has faced similiar issue

Re: Databricks execution failed with error state: InternalError, error message: failed to update run

jose_gonzalez — Tue, 18 Jul 2023 16:20:26 GMT

Are you able to check the driver's logs for more details?

Re: Databricks execution failed with error state: InternalError, error message: failed to update run

dataengineer17 — Wed, 19 Jul 2023 09:04:54 GMT

@jose_gonzalez yup I checked all driver logs (stdout, stderr, log4j) but nothing unusual

Re: Databricks execution failed with error state: InternalError, error message: failed to update run

saipujari_spark — Wed, 19 Jul 2023 19:52:45 GMT

@dataengineer17

It could be coming from the internal jobs service, If the issue persists I would recommend creating a support ticket.