szymon_dybczak
Esteemed Contributor III

Hi  @Niyojit ,

I think your proposed architecture is broadly the right direction, but I would separate lineage collection from lineage visualization.

Unity Catalog already exposes table-to-table relationships programmatically through system.access.table_lineage. It contains source_table_full_name and target_table_full_name, so it can effectively become the edge list for a custom lineage graph. 

For data quality, there are now several useful sources that can be joined to those lineage nodes:

  • Data Quality Monitoring / anomaly detection writes table-level results to system.data_quality_monitoring.table_results.This is especially useful because it already contains a consolidated health status plus freshness and completeness results. Interestingly, it also has a downstream_impact structure with the number of downstream tables and an impact/severity level, so Databricks itself is starting to connect DQ issues with dependency information.

  • Data profiling - the capability formerly called Lakehouse Monitoring - creates table_profile_metrics and  table_drift_metrics Delta tables. Those can provide things such as null counts, distributions, drift and custom metrics.

  • DQX can persist centralized summary metrics to a Delta table. It captures input/error/warning/valid row counts and also exposes per-check results through check_metrics, so you can derive things such as "8/10 checks passed" or identify the specific failed checks.

So, in short - this is achievableBuild graph edges from system.access.table_lineage, then extract information from DQ Monitoring + profiling + DQX into a dedicated table that you can later join to graph edges.

 

View solution in original post