Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Wednesday
Hi everyone,
I'm working on a client use case where they want to visualize data quality together with data lineage.
My current approach is to combine:
- Unity Catalog Data Lineage for upstream/downstream dependencies
- Lakehouse Monitoring (or other data monitoring capabilities)
- DQX (Data Quality Expectations) library for table-level data quality checks
The idea is to display the lineage graph and, for each table in the lineage, show the corresponding data quality metrics/check results (e.g., passed/failed expectations, freshness, completeness, etc.) directly alongside the table node.
I'm wondering:
- Has anyone implemented something similar in Databricks?
- Is it possible to enrich the lineage graph with DQX or monitoring results?
- Are there recommended APIs, system tables, or best practices for achieving this?
- Or is there a better architecture or approach to solve this use case?
I'd appreciate any suggestions or insights from the community.
Thanks in advance!
Niyojit