Read notebooks programmatically from databricks py...

evgvain · ‎12-20-2023

I need programmatically read the content of the specific notebook in my databricks workspace from another notebook in the same workspace. Following the databricks documentation, I extract the path to my notebook and then list all other notebooks in the directory. This works as expected.

notebook_path = os.path.dirname(dbutils.entry_point.getDbutils().notebook().getContext().notebookPath().getOrElse(None))
for f in dbutils.fs.ls(f"file:/Workspace{notebook_path}"):
    print(f)

For example: FileInfo(path='file:/Workspace/Users/***/Notebook1', name='Notebook1', size=0, modificationTime=0)

Then I try to read a specific notebook:

with open(f"/Workspace{notebook_path}/Notebook1", "r") as _f:
    c = _f.read()

I get the following error:

OSError: [Errno 95] Operation not supported

File /databricks/python/lib/python3.10/site-packages/IPython/core/interactiveshell.py:282, in _modified_open(file, *args, **kwargs)
    275 if file in {0, 1, 2}:
    276     raise ValueError(
    277         f"IPython won't let you open fd={file} by default "
    278         "as it is likely to crash IPython. If you know what you are doing, "
    279         "you can use builtins' open."
    280     )
--> 282 return io_open(file, *args, **kwargs)

evgvain · ‎12-20-2023

I was able to do this by using databricks.sdk.WorkspaceClient. However, I looks like a workaround. I would prefer to read the content of notebooks directly.

databricksURL = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiUrl().getOrElse(None)
myToken = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiToken().getOrElse(None)
w = WorkspaceClient(host=databricksURL, token=myToken)
export_response = w.workspace.export(path, format=ExportFormat.JUPYTER)

Any advice appriciated.

jenshumrich · ‎03-13-2024

Thanks a lot for sharing your code snippet. Helped me a lot to copy some notebooks into our fileshare.

from databricks.sdk import WorkspaceClient

from databricks.sdk.service import workspace

import base64

def notebook_reader(file_location😞

databricksURL = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiUrl().getOrElse(None)

myToken = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiToken().getOrElse(None)

w = WorkspaceClient(host=databricksURL, token=myToken)

export_response = w.workspace.export(file_location, format=workspace.ExportFormat.SOURCE)

return base64.b64decode(export_response.content)

Read notebooks programmatically from databricks python notebook