The error message you're encountering in PySpark when using dbutils.notebook.run() suggests that the context in which you are attempting to call the run() method is not valid. PySpark notebooks in Databricks have certain requirements when it comes to using dbutils.notebook.run() in a multi-threaded environment.

Here are some steps you can follow to address this issue:

Set Notebook Context:

• As indicated in the error message, you should set the notebook context using dbutils.notebook.setContext(ctx) before calling dbutils.notebook.run(). The ctx value should be retrieved from the main thread in the same cell.

Hello Kevin3,

In pyspark, the below line gets the context:
ctx = dbutils.notebook.entry_point.getDbutils().notebook().getContext()

But, this "dbutils.notebook.setContext(ctx)" doesn't set the Context()

I was searching the directory and found the method somewhat like this.

"Exception due to : Context not valid. If you are calling this outside the main thread, you must set the Notebook context via dbutils.notebook.setContext(ctx), where ctx is a value retrieved from the main thread (and the same cell) via dbutils.notebook.getContext()."

Maybe I am not using it the correct way. Let me know if you have a workaround for this situation:

1. Attaching the functions that I am using to run the child notebook.

2. I am running the function "execute_child_nb" in the stream microbatch.

Thanks,

Abhijit