topic Re: Import notebook with python script using API in Data Engineering

Import notebook with python script using API

RantoB — Fri, 12 Nov 2021 13:45:34 GMT

Hi,

I would like to import a python notebook to my databricks workspace from my local machine using a python script.

I manages to create the folder but then I have a status code 400 when I try to import a file :

create_folder = requests.post(
  '{}/api/2.0/workspace/mkdirs'.format(DBW_URL),
  headers={'Authorization': 'Bearer {}'.format(TOKEN)},
  json={"path": "/Repos/test"}
)
 
print(create_folder.status_code) # -> 200
 
python_code = """
# Databricks notebook source
print("This notebook has been imported via API.")
"""
 
data = base64.standard_b64encode(python_code.encode('utf-8')).decode('utf-8')
 
import_notebook = requests.post(
  '{}/api/2.0/workspace/import'.format(DBW_URL),
  headers={'Authorization': 'Bearer {}'.format(TOKEN)},
  json={
      "content": data,
      "path": "/Repos/test/hello.py",
      "language": "PYTHON",
      "overwrite": True,
      "format": "SOURCE"      
  }
)
 
print(import_notebook.status_code) # -> 400

I am not sure about the way I encoded the "content" value but I d'ont think this is the problem.

Thansk for your help.

Re: Import notebook with python script using API

cconnell — Fri, 12 Nov 2021 13:49:08 GMT

Where is this python code running, on your local machine or in Databricks?

Re: Import notebook with python script using API

RantoB — Fri, 12 Nov 2021 15:32:13 GMT

on my local machine.

Re: Import notebook with python script using API

Hubert-Dudek — Sun, 14 Nov 2021 17:30:44 GMT

you can make your life easier and use cli api:

pip install databricks-cli

and then:

from databricks_cli.workspace.api import WorkspaceApi
from databricks_cli.sdk.api_client import ApiClient
 
client = ApiClient(
    host='https://your.databricks-url.net',
    token=api_key
)
workspace_api = WorkspaceApi(client)
workspace_api.import_workspace(
    source_path="/your/dir/here/hello.py",
    target_path="/Repos/test/hello.py",
    overwrite=True
)

Re: Import notebook with python script using API

RantoB — Mon, 15 Nov 2021 08:13:15 GMT

Hi, Thanks for your answer.

Actually both your code and mine are working. However, I cannot write in the directory Repos which is reserved (but I can create subdirectories...)

Thanks to your code I got an error message which helped me to understand. With my code I had no error message.

Re: Import notebook with python script using API

Hubert-Dudek — Mon, 15 Nov 2021 15:46:15 GMT

any chance to be selected as the best answer 🙂 ? 😉

Re: Import notebook with python script using API

RantoB — Tue, 16 Nov 2021 13:33:21 GMT

Hi @Hubert Dudek ,

Do you know where I can find the documentation about the pythno api for databricks ?

Besides, do you know how to launch a job or notebook remotely with python api ?

Thanks

Re: Import notebook with python script using API

Hubert-Dudek — Tue, 16 Nov 2021 15:01:44 GMT

https://docs.databricks.com/dev-tools/cli/index.html

but I checked what is working there on github as cli docs are more for...cli not sdk https://github.com/databricks/databricks-cli/tree/master/databricks_cli/sdk

there is class JobsService and method run_now. I think is what you are looking for.

Re: Import notebook with python script using API

RantoB — Tue, 16 Nov 2021 15:06:03 GMT

I finally found. What I needed was there :

from databricks_cli.jobs.api import JobsApi
from databricks_cli.sdk.api_client import ApiClient

But I had to guess based on what you told me about databricks_cli.workspace.api

Re: Import notebook with python script using API

Ramya — Tue, 20 Sep 2022 04:40:38 GMT

Hi All,

I tried the same code but it's not working for me. I keep getting

import_workspace() got an unexpected keyword argument 'format'.

token_credential = DefaultAzureCredential()

scope = "2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default"

token = token_credential.get_token(scope)

access_token = str(token.token)

from databricks_cli.workspace.api import WorkspaceApi

from databricks_cli.sdk.api_client import ApiClient

client = ApiClient(

host='https://your.databricks-url.net',

token=access_token

)

workspace_api = WorkspaceApi(client)

workspace_api.import_workspace(

source_path="/home/hello.py",

target_path="/repo/test/hello.py",

language="PYTHON",

format = "SOURCE"

)

Re: Import notebook with python script using API

Ramya — Tue, 20 Sep 2022 19:31:53 GMT

Looks like the recent import_workspace api parameter is different. It worked after changed the correct parameter

workspace_api.import_workspace(

source_path="/home/hello.py",

target_path="/repo/test/hello.py",

language="PYTHON",

fmt="SOURCE",

is_overwrite = True

)