"command complete" but not executed

[jl1]

I have a notebook that calls other notebooks with `dbutils.notebook.run` and execute them as a 'Notebook job'. But sometimes when a notebook is taking a long time and the cluster is just waiting for, for instance, an api response, the subsequent commands that are submitted and shown as "waiting to run" just get a "command complete" message without being executed. It seems like the cluster waited (not inactive, as it checks with the api and see if there's a response) for a long time, but gave up on waiting at some point without any warning, and then terminated. the message "command complete" is too vague, and sometimes the notebook job does get executed. i haven't been able to reproduce this situation, it happens occasionally and coincide with the cluster having to wait around for a long time.

[Kaniz_Fatma]

Hi @jl1 , Based on the provided information, it seems like your issue might be related to the timeout settings of the dbutils.notebook.run command or the long waiting time for API responses.

The dbutils.notebook.run command has a timeoutSeconds parameter that controls the timeout of the run. If the called notebook does not finish running within the specified time, an exception is thrown. If the cluster is waiting for an API response for a long time, it might cause the notebook to exceed this timeout limit, leading to the termination of the command.

It would be better to consult Databricks support by filing a support ticket.