Read notebooks programmatically from databricks python notebook

evgvain
New Contributor II

‎12-20-2023 12:25 AM

I need programmatically read the content of the specific notebook in my databricks workspace from another notebook in the same workspace. Following the databricks documentation, I extract the path to my notebook and then list all other notebooks in the directory. This works as expected.

 

notebook_path = os.path.dirname(dbutils.entry_point.getDbutils().notebook().getContext().notebookPath().getOrElse(None))
for f in dbutils.fs.ls(f"file:/Workspace{notebook_path}"):
    print(f)

 

For example: FileInfo(path='file:/Workspace/Users/***/Notebook1', name='Notebook1', size=0, modificationTime=0)

Then I try to read a specific notebook:

 

with open(f"/Workspace{notebook_path}/Notebook1", "r") as _f:
    c = _f.read()

 

I get the following error:

 

OSError: [Errno 95] Operation not supported

File /databricks/python/lib/python3.10/site-packages/IPython/core/interactiveshell.py:282, in _modified_open(file, *args, **kwargs)
    275 if file in {0, 1, 2}:
    276     raise ValueError(
    277         f"IPython won't let you open fd={file} by default "
    278         "as it is likely to crash IPython. If you know what you are doing, "
    279         "you can use builtins' open."
    280     )
--> 282 return io_open(file, *args, **kwargs)

 

 

evgvain
New Contributor II

‎12-20-2023 02:29 AM

I was able to do this by using databricks.sdk.WorkspaceClient. However, I looks like a workaround. I would prefer to read the content of notebooks directly.

databricksURL = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiUrl().getOrElse(None)
myToken = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiToken().getOrElse(None)
w = WorkspaceClient(host=databricksURL, token=myToken)
export_response = w.workspace.export(path, format=ExportFormat.JUPYTER)

 Any advice appriciated. 

jenshumrich
Contributor
In response to evgvain

‎03-13-2024 03:28 AM

Thanks a lot for sharing your code snippet. Helped me a lot to copy some notebooks into our fileshare.
 
from databricks.sdk import WorkspaceClient
from databricks.sdk.service import workspace
import base64
 
def notebook_reader(file_location😞
    databricksURL = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiUrl().getOrElse(None)
    myToken = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiToken().getOrElse(None)
    w = WorkspaceClient(host=databricksURL, token=myToken)
    export_response = w.workspace.export(file_location, format=workspace.ExportFormat.SOURCE)
    return base64.b64decode(export_response.content)