cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

andrew0117
by Contributor
  • 1815 Views
  • 4 replies
  • 2 kudos

Resolved! master notebook cannot find the udf registered in the child notebook

The master notebook is calling a child notebook using dbutils.notebook.run("PathToChildnotebook"). The child notebook defines a user-defined function (UDF) and registers it using spark.udf.register. However, when the child notebook finishes running a...

  • 1815 Views
  • 4 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

@andrew li​ :The reason why the UDF cannot be found is that when the child notebook finishes running, the Spark context that was used to define and register the UDF is destroyed. Therefore, the UDF is no longer available in the Spark context used by ...

  • 2 kudos
3 More Replies
andrew0117
by Contributor
  • 1554 Views
  • 5 replies
  • 9 kudos

Resolved! How to call a few child notebooks from master notebook parallelly?

Planning using dbutils.notebook.run() to call all the child notebooks in the master notebook, but they are executed sequentially. 

  • 1554 Views
  • 5 replies
  • 9 kudos
Latest Reply
Kaniz
Community Manager
  • 9 kudos

Hi @andrew li​, We haven’t heard from you since the last response from @Uma Maheswara Rao Desula​, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please share it with the community, as it can be helpf...

  • 9 kudos
4 More Replies
Labels