topic Re: How to import a Lakeview Dashboard programmatically (API or CLI)? in Data Engineering

How to import a Lakeview Dashboard programmatically (API or CLI)?

Gutek — Fri, 16 Feb 2024 20:50:41 GMT

I'm trying to import a Lakeview Dashboard that I've originally exported through the CLI (version 0.213.0). The exported file has extension .lvdash.json and is a single line json file.

I can't get it to work, I tried this command:

databricks workspace import /Users/my@user.com/my_first_lakeview_dashboard.lvdash.json --profile prd --file dashboard.json

but I get: "Error: The zip file may not be valid or may be an unsupported version. Hint: Objects imported using format=SOURCE are expected to be zip encoded databricks source notebook(s) by default. Please specify a language using the --language flag if you are trying to import a single uncompressed notebook"

There is a language flag but the only options available are: [PYTHON, R, SCALA, SQL].

There is json flag but it's only for providing the request in JSON format.

There is no mention of Lakeview dashboards in this API. Is this even supported? Or do I have the syntax wrong? The Lakeview API only supports publishing.

Re: How to import a Lakeview Dashboard programmatically (API or CLI)?

miranda_luna_db — Sat, 17 Feb 2024 23:22:46 GMT

Thanks for flagging. There should be enhanced API documentation specific to Lakeview in the next week or two (PR is in review). Keep an eye out for a page called "Use the Lakeview API and Workspace API to create and manage Lakeview dashboards."

Currently, there is API support for Lakeview dashboard:

Export/Import (via workspace APIs)
List (via workspace APIs)
Get Status (via workspace APIs)
Publish

Additional areas are under development.

For a bit more information on Lakeview dashboard export/import via API specifically....

Exporting a dashboard

If we want to export the contents of mydashboard.lvdash.json, we can use the Workspace Export API to do so.

GET /api/2.0/workspace/export Query parameters: { "path": "/Users/my@user.com/examples/mydashboard.lvdash.json", "direct_download": true } Response: { "pages": [ { "name": "7db2d3cf", "displayName": "New Page" } ] }

This response shows the contents of a minimal dashboard definition which is blank.

Note that the file name without extension is what appears as the name of the dashboard in the Workspace (mydashboard.lvdash.json). If the direct_download property is left out of the request or is set to false, the response will include the base64 encoded version of the json string. We will use this later for the import request.

GET /api/2.0/workspace/export Query parameters: { "path": "/Users/my@user.com/examples/mydashboard.lvdash.json", "direct_download": false } Response: { "content": "eyJwYWdlcyI6W3sibmFtZSI6IjdkYjJkM2NmIiwiZGlzcGxheU5hbWUiOiJOZXcgUGFnZSJ9XX0=", "file_type": "lvdash.json" }

Importing a dashboard

If we want to import another dashboard into the workspace using the same contents as mydashboard.lvdash.json, we can do so by using the Workspace Import API. For the import to be properly recognized as a Lakeview dashboard, a few important parameters must be set:

"format": "AUTO" - this setting will allow the system to automatically detect the asset type.
"path": needs the file path to end in ".lvdash.json". This in conjunction with the format will mean the content will attempt to import as a Lakeview dashboard.

If these settings are not configured properly, the import might succeed but the dashboard would be treated like a normal file.

POST /api/2.0/workspace/import Request body parameters: { "path": "/Users/my@user.com/examples/myseconddashboard.lvdash.json", "content": "eyJwYWdlcyI6W3sibmFtZSI6IjdkYjJkM2NmIiwiZGlzcGxheU5hbWUiOiJOZXcgUGFnZSJ9XX0=", "format": "AUTO" } Response: {}

If we were to immediately issue the same API request, we would get an error:

{ "error_code": "RESOURCE_ALREADY_EXISTS", "message": "Path (/Users/my@user.com/examples/myseconddashboard.lvdash.json) already exists." }

In order to overwrite the contents of an existing dashboard in place, the "overwrite" property can be set:

Re: How to import a Lakeview Dashboard programmatically (API or CLI)?

yegorski — Wed, 29 May 2024 21:04:10 GMT

Thanks for the very detailed response. This is a fantastic example of a detailed support response.

We have a problem with the import API. No matter the combination of content parameter (base64, pure JSON, json dump, str, etc.) the API always returns

b'{"error_code":"MALFORMED_REQUEST","message":"Invalid JSON given in the body of the request - failed to parse given JSON"}\n'

Here's the code we're using: iterate over all our warehouses and create a dashboard for each. The JSON dashboard file is taken straight from

https://github.com/CodyAustinDavis/dbsql_sme/blob/main/Observability%20Dashboards%20and%20DBA%20Resources/Observability%20Lakeview%20Dashboard%20Templates/DBSQL%20Warehouse%20Advisor%20Observability%20Dashboard.lvdash.json

Has there been an update to the API to make Lakeview Dashboards work more smoothly?

Re: How to import a Lakeview Dashboard programmatically (API or CLI)?

yegorski — Thu, 30 May 2024 03:17:55 GMT

Turns out the problem was between keyboard and chair! The issues was with using requests data vs json. Here's the full working code:

import base64

import json

import os

import requests

WORKSPACE_DASHBOARDS_FOLDER = "/Workspace/Engineering Metrics/SQL Warehouse Monitoring"

with open("./dashboard_template.json", "r") as file:

DASHBOARD_TEMPLATE = json.load(file)

class Databricks:

def __init__(self, API_KEY):

self.url = "https://enigma-e2-premium.cloud.databricks.com/api/2.0"

self.api_key = API_KEY

self.headers = (

{

"Accept": "application/json",

"Content-Type": "application/json",

"Authorization": f"Authorization: Bearer {self.api_key}",

)

def make_request(self, method, query, **kwargs):

response = requests.request(

method, f"{self.url}{query}", headers=self.headers[0], **kwargs

)

response.raise_for_status()

return response

def list_serverless_warehouses(self):

response = self.make_request("get", f"/sql/warehouses").json()["warehouses"]

warehouses = []

for warehouse in response:

if warehouse["enable_serverless_compute"] == True:

warehouses.append(

{

"id": warehouse["id"],

"name": warehouse["name"],

}

)

return warehouses

def create_dashboard(self, warehouse_id: str, warehouse_name: str):

dashboard_config = DASHBOARD_TEMPLATE

for i, v in enumerate(dashboard_config["datasets"]):

dashboard_config["datasets"][i]["parameters"][0]["defaultSelection"][

"values"

]["values"][0]["value"] = warehouse_id

request_data = {

"path": f"{WORKSPACE_DASHBOARDS_FOLDER}/{warehouse_name}.lvdash.json",

"content": base64.b64encode(

json.dumps(dashboard_config).encode("utf-8")

).decode("utf-8"),

"format": "AUTO",

"overwrite": "true",

}

return self.make_request("post", f"/workspace/import", json=request_data)

databricks = Databricks(os.getenv("DATABRICKS_TOKEN_PREMIUM"))

warehouses = databricks.list_serverless_warehouses()

for warehouse in warehouses:

databricks.create_dashboard(warehouse["id"], warehouse["name"])

Re: How to import a Lakeview Dashboard programmatically (API or CLI)?

miranda_luna_db — Thu, 30 May 2024 13:23:12 GMT

Glad you've got everything up and running!