Integrating Unity Catalog ML Models into Data Lineage

[23940829381]

I saw a really nice article (https://www.databricks.com/blog/announcing-public-preview-volumes-databricks-unity-catalog) on the incorporation of various elements of data lineage within Unity Catalog. In my own exploration, I've been able to replicate graphs linking volumes to tables and tables to tables, but I have not been able to achieve the attached example, where a model is tied into the lineage graph.

When I register models in Unity Catalog and perform transformations with them, no linkage shows up to the model itself ever. Further, if I look at a registered model within Unity Catalog, it does show the graph element for the model itself as in the attached photo, but it is linked neither to training/ingest or predictive output tables.

Has anyone been able to replicate the attached image? Thank you!

[23940829381]

Hi @Retired_mod -- thank you for your response! I wanted to confirm that we had been able to see the lineage graph no problem for all non-ML volume/table transformations. In addition each time we retrained models and registered a best performer, we could see its versioning lineage in Unity Catalog and the versions themselves were each linked back to MLflow runs as you pointed out. All of the models and data we were using are far less than a year old, so based on the third point, I am still surprised that they didn't show up in the lineage. Do you have any other suggestions? Thank you!