cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Warehousing & Analytics
Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Community. Share insights, tips, and best practices for leveraging data for informed decision-making.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Data Quality Dashbaord - I have a requirement to build a custom designed quality dashboard in dbx

Databricks2206
New Contributor

I have a requirement to build a custom designed quality dashboard in dbx. The requirement is like user have to view the quality of data metrics in a single dahsboard for many tables, irrespective of viewing it via inbuilt data quality option available in every table. what is the best design approcah .please explain.

1 REPLY 1

DoTA
Contributor

Hi @Databricks2206

For us we have try a couple of approaches

Option 1: DQX (Databricks Labs Data Quality Framework) - this is purpose-built for what you're describing. It ships a pre-built dashboard with three tabs, including a "Data Quality Summary" tab that gives aggregated quality statistics across all monitored tables, giving you a high-level view of your data quality status - plus a time-series tab and a full-snapshot tab per table. It deploys automatically when you install DQX via the Databricks CLI, or you can import the dashboard directly from the GitHub repo. See: https://databrickslabs.github.io/dqx/docs/guide/quality_dashboard/

 

Option 2: Native Lakehouse Monitoring - if you'd rather not add a labs/third-party framework, Databricks' built-in monitoring creates two Delta tables per monitored table: {output_schema}.{table_name}_profile_metrics and {output_schema}.{table_name}_drift_metrics (docs: https://docs.databricks.com/aws/en/lakehouse-monitoring/monitor-output). If you point every table's monitor at the same output_schema_name, all your tables' metrics land in one schema, namespaced by table name - so you can build a single UNION ALL view across them and put a regular AI/BI Dashboard on top. More manual than DQX's dashboard, but zero extra dependencies.

 

For "many tables, one view" specifically, I'd start with DQX's dashboard since that's exactly the gap it's built to fill - worth checking if the Data Quality Summary tab already covers your metrics before building something custom on Lakehouse Monitoring.

 

We are now basically forking the DQX, and customizing it ourself.