topic Re: Run a Databricks notebook from another notebook with ipywidget in Machine Learning

Run a Databricks notebook from another notebook with ipywidget

mbejarano89 — Thu, 23 Mar 2023 20:32:07 GMT

I am trying to run a notebook from another notebook using the dbutils.notebook.run as follows:

import ipywidgets as widgets

from ipywidgets import interact

from ipywidgets import Box

button = widgets.Button(description='Run model')

out = widgets.Output()

def on_button_clicked(b):

button.description = 'Run model'

with out:

dbutils.notebook.run("/mynotebookpath",60)

button.on_click(on_button_clicked)

widgets.VBox([button, out])

However, I am getting the following error:

IllegalArgumentException: Context not valid. If you are calling this outside the main thread, you must set the Notebook context via dbutils.notebook.setContext(ctx), where ctx is a value retrieved from the main thread (and the same cell)

I can run the notebook just fine when I do

%run

on a single cell and even

dbutils.notebook.run("/mynotebook", 60)

on a single cell. However I cannot get it to run within the ipywidget context

Re: Run a Databricks notebook from another notebook with ipywidget

Anonymous — Sat, 25 Mar 2023 06:47:02 GMT

@Marcela Bejarano :

The error you are seeing is because the dbutils module requires a valid notebook context to run the notebook. When running code from an ipywidget context, the main thread and the widget thread are different. To solve this, you can try setting the notebook context before calling dbutils.notebook.run()

import ipywidgets as widgets
from ipywidgets import interact, Box
from pyspark.sql import SparkSession
 
button = widgets.Button(description='Run model')
out = widgets.Output()
 
def on_button_clicked(b):
    button.description = 'Run model'
    with out:
        # Get the current notebook context
        current_notebook = get_ipython().get_parent()
        # Set the notebook context for dbutils
        dbutils.notebook.setContext(current_notebook)
        # Run the notebook
        dbutils.notebook.run("/mynotebookpath",60)
 
button.on_click(on_button_clicked)
widgets.VBox([button, out])

This should set the notebook context correctly and allow you to run the notebook from the ipywidget context.

Re: Run a Databricks notebook from another notebook with ipywidget

mbejarano89 — Sat, 25 Mar 2023 09:44:08 GMT

Hello,

Thank you for this. I tried running it but now I am getting the error:

AttributeError: 'NotebookHandler' object has no attribute 'setContext'

Re: Run a Databricks notebook from another notebook with ipywidget

Anonymous — Thu, 30 Mar 2023 05:38:59 GMT

Hi @Marcela Bejarano

I'm sorry you could not find a solution to your problem in the answers provided.

Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.

I suggest providing more information about your problem, such as specific error messages, error logs or details about the steps you have taken. This can help our community members better understand the issue and provide more targeted solutions.

Alternatively, you can consider contacting the support team for your product or service. They may be able to provide additional assistance or escalate the issue to the appropriate section for further investigation.

Thank you for your patience and understanding, and please let us know if there is anything else we can do to assist you.

Re: Run a Databricks notebook from another notebook with ipywidget

TheDarkKnight — Tue, 29 Aug 2023 11:54:38 GMT

Hey !

Have you found a solution for that isssue?
I am facing the same problem...
"AttributeError: 'NotebookHandler' object has no attribute 'setContext'"

Thanks in advance!

Re: Run a Databricks notebook from another notebook with ipywidget

Sreekanth_N — Tue, 29 Aug 2023 14:11:14 GMT

I am also getting the same error

Re: Run a Databricks notebook from another notebook with ipywidget

Sreekanth_N — Tue, 29 Aug 2023 14:16:32 GMT

As I could see the pyspark stream is not supporting this setContext, ideally it should have alternative approach. please suggest what is approach where pyspark stream is internally calling to another notebook parallel