topic Re: Attach notebook to cluster via REST API in Data Engineering

Attach notebook to cluster via REST API

akihiko — Fri, 24 Feb 2023 07:47:37 GMT

Is it possible to attach a notebook to cluster and run it via the REST API?

The closest approach I have found is to run a notebook, export the results (HTML!) and import it into the workspace again, but this does not allow us to retain the original execution context. The 1.2 API allows for the creation and manipulation of execution contexts, but I can't figure out how to actual attach a notebook.

My use-case is that we have notebook templates for security incidents that contain some pre-defined cells/queries. We want the system to run these queries before handing the notebook off to an analyst where she can do further analysis.

Re: Attach notebook to cluster via REST API

Ajay-Pandey — Fri, 24 Feb 2023 08:17:31 GMT

Hi @Akihiko Nagata ,

I think for now we don't have such option available through databricks API.

Re: Attach notebook to cluster via REST API

Vivian_Wilfred — Fri, 24 Feb 2023 14:33:24 GMT

Hi @Akihiko Nagata , have you checked the jobs API? You can run a job on the existing cluster that can use the notebook of concern. I believe this is the only way.

https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsRunsSubmit

Re: Attach notebook to cluster via REST API

akihiko — Mon, 27 Feb 2023 12:44:18 GMT

Hi @Vivian Wilfred, I did try that, but that won't allow you to keep the execution context, so it only partially works.

I guess for now I will use this API and export the executed notebook.

Re: Attach notebook to cluster via REST API

baert23 — Tue, 27 Aug 2024 21:07:38 GMT

I'm looking for a way to programmatically copy a notebook in Databricks using the workspace/export and workspace/import APIs. Once the notebook is copied, I want to automatically attach it to a specific cluster using its cluster ID. The challenge is that I can't use Jobs because the notebook contains widgets that require user interaction.

Is there a way to achieve this, or does anyone know of a workaround to programmatically attach a notebook to a cluster after it has been imported?