Deploying Overwatch on Databricks (AWS) with System Tables as the Data Source

jiteshraut20 — Wed, 14 Aug 2024 18:37:04 GMT

Introduction

Overwatch is a powerful tool for monitoring and analyzing your Databricks environment, providing insights into resource utilization, cost management, and system performance. By leveraging system tables as the data source, you can gain a comprehensive view of all workspaces in your Databricks account. This article will guide you through the process of deploying Overwatch on Databricks (AWS), using system tables as the primary data source.

Official Documentation: General overview and guidance for Overwatch and Overwatch Implementation, though it may not include the latest updates.

Data Dictionary: Detailed definitions of the data elements available after successful deployment of Overwatch for consumption.

Architecture: Reference architecture for deploying Overwatch on AWS.

Prerequisites

1. Enable System Table Schemas

Before deploying Overwatch, you must enable system table schemas. These schemas are governed by Unity Catalog, and you must have at least one Unity Catalog-enabled workspace in your account to access system tables. The system tables aggregate data from all workspaces in your account but can only be accessed from a Unity Catalog-enabled workspace.

Key Points:

- System Table Governance: Unity Catalog governs system tables.

- Access: Only accessible from Unity Catalog-enabled workspaces.

- Enabling Schemas: System schemas must be enabled at the schema level, and an account admin must manually enable them.

How to Enable System Table Schemas:

- List available system schemas using the following `curl` command:

curl -v -X GET -H “Authorization: Bearer <PAT Token>” “https://<workspace>.cloud.databricks.com/api/2.0/unity-catalog/metastores/<metastore-id>/systemschemas”

- Enable a system schema using the following `curl` command:

curl -v -X PUT -H “Authorization: Bearer <PAT Token>” “https://<workspace>.databricks.com/api/2.0/unity-catalog/metastores/<metastore-id>/systemschemas/<SCHEMA_NAME>”

For more details, refer to the official Databricks documentation: Enable System Table Schemas.

2. Generate a Personal Access Token (PAT)

To interact with the Databricks API, you need to generate a Personal Access Token (PAT).

Steps to Generate a PAT:

a. Log in to Databricks: Access your Databricks workspace using your credentials.

b. Access User Settings: Click on your profile icon and select “User Settings.”

c. Generate a New Token: Under the “Access Tokens” section, click “Generate New Token.”

d. Configure the Token: Set a lifetime and optionally provide a description.

e. Save the Token: Copy the token immediately, as it will not be displayed again.

3. Securely Store the PAT in a Secret Scope

To securely store the PAT, create a secret scope using the Databricks CLI.

Steps to Create a Secret Scope:

a. Set Up the Databricks CLI: Install and configure the CLI with your workspace URL and PAT.

pip install databricks-cli

databricks configure — token

b. Create a Secret Scope: Use the following command to create a scope:

databricks secrets create-scope — scope <scope-name>

c. Add the PAT to the Secret Scope: Store the PAT in the secret scope.

databricks secrets put — scope <scope-name> — key <secret-key>

Configuring Overwatch with System Tables

To deploy Overwatch using system tables as the data source, you’ll need to create a configuration file that contains the necessary details for your Databricks environment. Below is a sample schema for the configuration file with sample values:

workspace_name : Workspace-TestOW

workspace_id : 1857915041000000

workspace_url : https://dbc-664fbb00-0000.cloud.databricks.com

api_url : https://mumbai.cloud.databricks.com

cloud : AWS

primordial_date : 28–07–2024

storage_prefix : /mnt/overwatch_global

etl_database_name : overwatch_etl

consumer_database_name : overwatch_global

secret_scope : secret_scope_for_overwatch

secret_key_dbpat : dapicd0c0c0000e0000a0a0d0c0f0f0e000e

auditlogprefix_source_path : system

eh_name :

eh_scope_key :

interactive_dbu_price : 0.55

automated_dbu_price : 0.3

sql_compute_dbu_price : 0.22

jobs_light_dbu_price : 0.1

max_days : 30

excluded_scopes :

active : TRUE

proxy_host :

proxy_port :

proxy_user_name :

proxy_password_scope :

proxy_password_key :

success_batch_size :

error_batch_size :

enable_unsafe_SSL :

thread_pool_size :

api_waiting_time :

Save this configuration as csv and save/ upload it to DBFS (we will need path of this file later in further deployment steps)

workspace_name,workspace_id,workspace_url,api_url,cloud,primordial_date,storage_prefix,etl_database_name,consumer_database_name,secret_scope,secret_key_dbpat,auditlogprefix_source_path,eh_name,eh_scope_key,interactive_dbu_price,automated_dbu_price,sql_compute_dbu_price,jobs_light_dbu_price,max_days,excluded_scopes,active,proxy_host,proxy_port,proxy_user_name,proxy_password_scope,proxy_password_key,success_batch_size,error_batch_size,enable_unsafe_SSL,thread_pool_size,api_waiting_time
Workspace-TestOW,1857915041000000,https://dbc-664fbb00-0000.cloud.databricks.com,https://mumbai.cloud.databricks.com,AWS,28-07-2024,/mnt/overwatch_global,overwatch_etl,overwatch_global,secret_scope_for_overwatch,dapicd0c0c0000e0000a0a0d0c0f0f0e000e,system,,,0.55,0.3,0.22,0.1,30,,TRUE,,,,,,,,,,

Configurations: Configuration settings and options for deploying Overwatch.

Custom Cost: Information on configuring custom cost settings in Overwatch.

Optimal Databricks Configuration

For deploying Overwatch, the following configuration worked optimally:

- Cluster DBR Version: 13.3 LTS

- Overwatch JAR Version: 0.8.1.2 (Latest as of August 2024)

This specific combination of DBR and Overwatch JAR is recommended, as it proved to be the most reliable in recent tests. Other versions may encounter compatibility issues.

Missing Libraries:

In the 13.3 LTS DBR version, a couple of libraries were missing:

- org.scalaj:scalaj-http_2.12:2.4.2

- dataframe_rules_engine_2.12:0.2.0

These libraries might need to be installed manually on the cluster to ensure full functionality.

Steps to Deploy Overwatch on Databricks (AWS)

Here is a comprehensive guide to deploying Overwatch on Databricks using the provided setup and configuration parameters:

1. Import Library

Begin by importing the necessary library for Overwatch deployment:

import com.databricks.labs.overwatch.MultiWorkspaceDeployment

2. Set Configuration Parameters

Define the parameters required for the deployment:


val PATHTOCSVCONFIG = "dbfs:/FileStore/overwatch/configs/overwatch Configuration.csv"
val CONFIGTABLENAME = "overwatch_dev_config"
val TEMPDIR = "dbfs:/mnt/overwatch/tmp" // Update as necessary
val PARALLELISM = 1 // Adjust based on the number of workspaces to deploy
val STORAGEPREFIX = "dbfs:/mnt" // Update as necessary

- PATHTOCSVCONFIG: Path to the CSV file with Overwatch configuration.

- CONFIGTABLENAME: Delta table where the Overwatch configuration will be saved or updated.

- TEMPDIR: Temporary directory for intermediate files during deployment.

- PARALLELISM: Number of workspaces to load simultaneously (up to ~20).

- STORAGEPREFIX: Prefix for storage paths.

3. Save CSV Content to Configuration Table

Load the CSV file and save its contents to the configuration table:

spark.read
.option("header", "true")
.option("ignoreLeadingWhiteSpace", true)
.option("ignoreTrailingWhiteSpace", true)
.csv(PATHTOCSVCONFIG)
.coalesce(1)
.write
.option("mergeSchema", "true")
.mode("overwrite")
.format("delta")
.saveAsTable(CONFIGTABLENAME)

4. Validate Deployment Configuration

Validate the deployment configuration using the `MultiWorkspaceDeployment` class:

MultiWorkspaceDeployment(CONFIGTABLENAME, TEMPDIR).validate(PARALLELISM)

Validation: Guidance on validating your Overwatch deployment.

5. Review Validation Results

Check the validation results to ensure everything is configured correctly:

import org.apache.spark.sql.expressions.Window
import org.apache.spark.sql.functions._
// Display the latest validation results
val windowSpec = Window.orderBy(col("snapTS").desc)
display(
spark.read.format("delta")
.load("dbfs:/mnt/overwatch_global_d1/report/validationReport")
.withColumn("rank", rank().over(windowSpec))
.filter(col("rank") === 1)
.drop("rank")
.orderBy("validated")
)

6. Final Deployment

Perform the final deployment for each layer (Bronze, Silver, Gold):

Bronze Layer:

MultiWorkspaceDeployment(CONFIGTABLENAME, TEMPDIR).deploy(PARALLELISM, "Bronze")

Silver Layer:

MultiWorkspaceDeployment(CONFIGTABLENAME, TEMPDIR).deploy(PARALLELISM, "Silver")

Gold Layer:

MultiWorkspaceDeployment(CONFIGTABLENAME, TEMPDIR).deploy(PARALLELISM, "Gold")

Running as Overwatch: Additional details on running Overwatch.

Final Deployment Report

Check the final deployment report using:

SELECT * FROM overwatch_etl.pipReport ORDER BY Pipeline_SnapTs DESC

By following these steps, you will deploy Overwatch on Databricks efficiently, using system tables as your data source and ensuring that all layers of the deployment are properly configured and validated.

Got it! If you need any further assistance or have additional questions, feel free to reach out at contact@jiteshraut.me or connect with you on LinkedIn.

Thank you!

Re: Deploying Overwatch on Databricks (AWS) with System Tables as the Data Source

AmanJain — Thu, 15 Aug 2024 06:30:24 GMT

hey @jiteshraut20

The below mentioned libraries are not required for the deployment. Please read the documentation carefully

- org.scalaj:scalaj-http_2.12:2.4.2

- dataframe_rules_engine_2.12:0.2.0

Also all the overwatch documents are updated frequently after every release.

Re: Deploying Overwatch on Databricks (AWS) with System Tables as the Data Source

raghu2 — Wed, 08 Jan 2025 23:09:11 GMT

hi @jiteshraut20, Thanks for your post. From my set up, validation seems to work.

Wrote 32 bytes. Validation report has been saved to dbfs:/mnt/overwatch_global/multi_ow_dep/report/validationReport
Validation report details
Total validation count: 35
Failed validation count:0
Report run duration in sec : 14

Deployment step is resulting in this error:

AnalysisException: CREATE SCHEMA in Unity Catalog must use MANAGED LOCATION, not LOCATION

Can you please let me know correct value for :

val STORAGEPREFIX = "dbfs:/mnt/overwatch_global/multi_ow_dep" // CHANGEME

Thanks,

topic Deploying Overwatch on Databricks (AWS) with System Tables as the Data Source in Data Engineering

Deploying Overwatch on Databricks (AWS) with System Tables as the Data Source

Introduction

Prerequisites

Re: Deploying Overwatch on Databricks (AWS) with System Tables as the Data Source

Re: Deploying Overwatch on Databricks (AWS) with System Tables as the Data Source