topic How can I programmatically get my notebook path? in Data Engineering

How can I programmatically get my notebook path?

User16752241457 — Thu, 10 Jun 2021 12:19:55 GMT

I'm writing some code that trains a ML model using MLflow and a given set of hyperparameters. This code is going to be run by several folks on my team and I want to make sure that the experiment that get's created is created in the same directory as the notebook - i.e. if someone clones the notebook into their own user folder, the MLflow experiment should be pointed to their notebooks new location.

Re: How can I programmatically get my notebook path?

Ryan_Chynoweth — Mon, 21 Jun 2021 19:38:48 GMT

In Scala you can use the "dbutils.notebook.getContext.notebookPath" function. If needed for Python or R, the only way to share this value would be to use widgets. Run the code with %scala, pass the value into a widget, then read the widget in Python or R.

Re: How can I programmatically get my notebook path?

User16857281974 — Fri, 30 Jul 2021 19:51:33 GMT

In Scala the call is

dbutils.notebook.getContext.notebookPath.get

In Python the call is

dbutils.entry_point.getDbutils().notebook().getContext().notebookPath().getOrElse(None)

If you need it in another language, a common practice would be to pass it through spark config.

Ignoring that we can get the value in Python (as seen above), if you start with a Scala cell like this:

%scala
val path = dbutils.notebook.getContext.notebookPath.get
spark.conf.set("com.whaterver.notebook-path")

And then in the next Python cell (the same concepts applies to R and SQL)

%python
path = spark.conf.get("com.whaterver.notebook-path")

You just want to adjust the configuration parameter's name to be a unique value so as not to accidentally clash with something. For example, we often use "com.databricks.training"