- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
yesterday
Hi @Niyojit ,
I think your proposed architecture is broadly the right direction, but I would separate lineage collection from lineage visualization.
Unity Catalog already exposes table-to-table relationships programmatically through system.access.table_lineage. It contains source_table_full_name and target_table_full_name, so it can effectively become the edge list for a custom lineage graph.
For data quality, there are now several useful sources that can be joined to those lineage nodes:
- Data Quality Monitoring / anomaly detection writes table-level results to system.data_quality_monitoring.table_results.This is especially useful because it already contains a consolidated health status plus freshness and completeness results. Interestingly, it also has a downstream_impact structure with the number of downstream tables and an impact/severity level, so Databricks itself is starting to connect DQ issues with dependency information.
- Data profiling - the capability formerly called Lakehouse Monitoring - creates table_profile_metrics and table_drift_metrics Delta tables. Those can provide things such as null counts, distributions, drift and custom metrics.
- DQX can persist centralized summary metrics to a Delta table. It captures input/error/warning/valid row counts and also exposes per-check results through check_metrics, so you can derive things such as "8/10 checks passed" or identify the specific failed checks.
So, in short - this is achievable. Build graph edges from system.access.table_lineage, then extract information from DQ Monitoring + profiling + DQX into a dedicated table that you can later join to graph edges.