- 2992 Views
- 4 replies
- 2 kudos
Resolved! master notebook cannot find the udf registered in the child notebook
The master notebook is calling a child notebook using dbutils.notebook.run("PathToChildnotebook"). The child notebook defines a user-defined function (UDF) and registers it using spark.udf.register. However, when the child notebook finishes running a...
- 2992 Views
- 4 replies
- 2 kudos
- 2 kudos
@andrew li​ :The reason why the UDF cannot be found is that when the child notebook finishes running, the Spark context that was used to define and register the UDF is destroyed. Therefore, the UDF is no longer available in the Spark context used by ...
- 2 kudos
- 2523 Views
- 4 replies
- 9 kudos
Resolved! How to call a few child notebooks from master notebook parallelly?
Planning using dbutils.notebook.run() to call all the child notebooks in the master notebook, but they are executed sequentially.
- 2523 Views
- 4 replies
- 9 kudos
- 9 kudos
Hi @andrew li​ Yes, They do run on the cluster on which the master notebook is running. Specifically, we are multithreading the Spark driver with Futures to enable parallel job submission. You can check out more on threads and futures for deeper unde...
- 9 kudos