Geting error Caused by: com.databricks.NotebookExecutionException: FAILED

DipakBachhav
New Contributor III

I am trying to run the below notebook through databricks but getting the below error. I have tried to update the notebook timeout and the retry mechanism but still no luck yet. 

  NotebookData("/Users/mynotebook",9900, retry=3)

   ]

   res = parallelNotebooks(notebooks, 2)

   result = [f.result(timeout=9900) for f in res] # This is a blocking call.

   print(result)

Can someone please help me to sort out this issue? Thanks

  %python

   from concurrent.futures import ThreadPoolExecutor

   class NotebookData:

    def __init__(self, path, timeout, parameters=None, retry=0):

     self.path = path

     self.timeout = timeout

     self.parameters = parameters

     self.retry = retry

   def submitNotebook(notebook):

    print("Running notebook %s" % notebook.path)

    try:

     if (notebook.parameters):

      return dbutils.notebook.run(notebook.path, notebook.timeout, notebook.parameters)

     else:

      return dbutils.notebook.run(notebook.path, notebook.timeout)

    except Exception:

     if notebook.retry < 1:

      raise

     print("Retrying notebook %s" % notebook.path)

     notebook.retry = notebook.retry - 1

     submitNotebook(notebook)

   def parallelNotebooks(notebooks, numInParallel):

    

    # This code limits the number of parallel notebooks.

    with ThreadPoolExecutor(max_workers=numInParallel) as ec:

     return [ec.submit(submitNotebook, notebook) for notebook in notebooks]

   notebooks = [    

   

    NotebookData("/Users/mynotebook",1200000, retry=0)

   ]

   res = parallelNotebooks(notebooks, 2)

   result = [f.result(timeout=1200000) for f in res] # This is a blocking call.

   print(result)

Error: 

  Py4JJavaError: An error occurred while calling o1741._run.

  : com.databricks.WorkflowException: com.databricks.NotebookExecutionException: FAILED

   at com.databricks.workflow.WorkflowDriver.run(WorkflowDriver.scala:95)

   at com.databricks.dbutils_v1.impl.NotebookUtilsImpl.run(NotebookUtilsImpl.scala:122)

   at com.databricks.dbutils_v1.impl.NotebookUtilsImpl._run(NotebookUtilsImpl.scala:89)

   at sun.reflect.GeneratedMethodAccessor820.invoke(Unknown Source)

   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

   at java.lang.reflect.Method.invoke(Method.java:498)

   at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)

   at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)

   at py4j.Gateway.invoke(Gateway.java:295)

   at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)

   at py4j.commands.CallCommand.execute(CallCommand.java:79)

   at py4j.GatewayConnection.run(GatewayConnection.java:251)

   at java.lang.Thread.run(Thread.java:748)

  Caused by: com.databricks.NotebookExecutionException: FAILED

   at com.databricks.workflow.WorkflowDriver.run0(WorkflowDriver.scala:141)

   at com.databricks.workflow.WorkflowDriver.run(WorkflowDriver.scala:90)

   ... 12 more