cancel
Showing results for 
Search instead for 
Did you mean: 
Technical Blog
Explore in-depth articles, tutorials, and insights on data analytics and machine learning in the Databricks Technical Blog. Stay updated on industry trends, best practices, and advanced techniques.
cancel
Showing results for 
Search instead for 
Did you mean: 
MohanaBasak
Databricks Employee
Databricks Employee

As organizations continue to scale their data infrastructure, efficient resource utilization, cost control, and operational transparency are paramount for success. With the growing adoption of Databricks, monitoring and optimizing compute usage and data processes are essential. To help address these needs, I’ve developed a suite of Databricks dashboards designed to provide clear visibility into cost, job performance, and data lineage insights—empowering data teams to make informed decisions.

You can explore the code and setup instructions for these dashboards in my GitHub repository Databricks Dashboard Suite.

Note: These dashboards are independently developed and are not officially supported by Databricks.

Databricks offers built-in usage dashboards that provide cost breakdowns by product, SKU, and tags, along with analyses of high-cost sources. The product team is continually enhancing these dashboards to improve the user experience. Complementing these, the custom dashboards in this project offer additional granular metrics, comprehensive documentation, and advanced customization options to support detailed cost and performance tracking.

Background: The Need for Unified Cost and Operational Tracking

Databricks offers immense flexibility to perform workloads, but this can also bring complexity in tracking various costs and operational metrics. With workloads spanning SQL, job clusters, and notebooks, the ability to monitor these aspects in one place becomes critical. This leaves data teams in need of customized dashboards that span all activities across the platform.

Note: Certain parts of the video appear blank because they have been intentionally masked to protect sensitive information. Actual values can be seen once you install and configure the dashboards in your environment.

The dashboard suite was designed with this goal in mind. Built using system tables, SQL queries, and Databricks' powerful visualization tools, it brings a cohesive view of usage, cost, and governance under one umbrella.

Screenshot 2024-12-15 at 5.20.05 PM.png

Project Overview

This project consists of four core dashboards, each addressing a specific operational need. Together, they provide a holistic view of Databricks’ operational insights:

  1. Databricks Unified Cost Analysis Dashboard: Focuses on cost trends across all compute types (e.g., all-purpose clusters, jobs, Delta Live Tables, model inference). Visualizations include daily spend per compute type, weekly and monthly cost trends, and per-workspace costs.Screenshot 2024-12-15 at 4.56.02 PM.png

  2. Job Operations and Cost Management Dashboard: Provides cost insights and operational metrics for job workloads. It includes team-level cost analysis, visualizations for job counts and runtimes, failure rates, resource usage, and cluster performance.
    Screenshot 2024-12-15 at 5.11.08 PM.png

  3. DBSQL Cost & Query Performance Dashboard: Highlights SQL workload costs and query-level performance. This dashboard provides visualizations on daily costs per workspace and SKU, a cost breakdown by teams and users, and a detailed query performance analysis with specific metrics like runtime and queue time. The cost per query calculation in this dashboard is based on the method shared by Cody Davis in his blog The New Databricks SQL Warehouse Advisor Dashboard, which serves as an example of calculating query costs. Note that there are multiple ways to calculate query costs, depending on the specific metrics you want to measure. In the future, if query costs are directly added to the system tables, this project will be updated to leverage that data for more accurate and streamlined calculations.Screenshot 2024-12-15 at 5.06.35 PM.png

  4. Data Lineage and Catalog Utilization Dashboard: Built from the Unity Catalog lineage table, this dashboard shows data movement across Databricks environments. Visualizations cover entity access over time, top catalog usage, and detailed table-level lineage to help teams track how data flows across jobs, queries, and dashboards.Screenshot 2024-12-15 at 5.01.17 PM.png

Each dashboard is equipped with configurable filters, allowing users to narrow down by workspace, date range, compute type, job ID, or team, making it easy to surface the most relevant information at any given time. Additionally, every section of the dashboards includes detailed documentation at the beginning, helping users understand the visualizations and their context. To further enhance navigation, the first dashboard at the top contains an index section with links to the other dashboards in the suite, enabling seamless access to deeper insights.

Screenshot 2024-12-15 at 5.30.48 PM.png

Key Features and Benefits

  1. Cost Efficiency: By providing cost breakdowns by compute type, workspace, and team, these dashboards enable users to pinpoint high-cost areas. Teams can then take informed actions to optimize their usage and reduce unnecessary expenses.

  2. Operational Insights: Detailed metrics on query performance, job runtimes, failure rates, and resource allocation make it easier to identify and resolve bottlenecks, ensuring smoother workflows across Databricks.

  3. Data Lineage Transparency: The lineage-focused dashboard offers valuable insights into how data flows across different entities within Databricks, enhancing data governance and enabling better tracking of data dependencies.

  4. Customizable Deployment: The dashboards are easy to deploy via a single parameterized notebook, which creates necessary tables and functions, configures filters, and generates the visualization setup, allowing for easy customization to match specific organizational needs.

Implementation: Getting Started with the Dashboards

The dashboard suite is packaged in a GitHub repository, making deployment straightforward. Users need only to clone the repository using Git Folders and run the setup notebook (detailed instructions here). This automatically deploys all dashboards and prepares the necessary system tables and functions.

Note: The setup notebook includes optional configuration parameters, such as tags for team-based breakdowns, and options to fetch workspace names for clearer visualizations.

The dashboards will be modified as Databricks adds more features, such as additional system tables. Regular updates ensure that the dashboards remain aligned with Databricks' evolving ecosystem.

In Summary

For organizations utilizing Databricks, having clear insights into cost and performance metrics is essential for long-term success. The Databricks Custom Dashboards Project brings together powerful cost, operational, and lineage insights into one accessible suite, enabling teams to make data-driven decisions efficiently. As Databricks capabilities expand, these dashboards will continue to evolve, providing a robust toolset for maximizing value from your Databricks environment.

If you’re interested in giving these dashboards a try, or contributing to the project, feel free to check out the GitHub repository here and explore the capabilities that can help make your Databricks operations more transparent, efficient, and cost-effective.

Disclaimer

This project and the accompanying dashboards are not official Databricks products. They are community-built resources provided as-is, with no dedicated ongoing support. Please review and test in your environment, as they are intended for use at your own risk.

2 Comments