Hello!

We got some pipelines running in Azure Data Factory that call Databricks Notebooks to run data transformations. This morning at 6:21 AM (UTC) we got an error " Unexpected failure while fetching notebook" inside a notebook that calls another one.

Upon checking the Driver Logs for that job in particular, in the "Standard Error" it appears:

OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
ANTLR Tool version 4.7 used for code generation does not match the current runtime version 4.8

And then some lines below the initial error

Py4JJavaError: An error occurred while calling o1771._run.
: com.databricks.WorkflowException: com.databricks.NotebookExecutionException: FAILED: Unexpected failure while fetching notebook.

The work around I got was to rerun the pipeline and it succeeded but I'm still not sure what happened, if it's something from our side, is there something I can do to prevent this from happening at again?