topic Re: Lineage between model and source code breaks on movement of source notebook. How to rectify it? in Data Engineering

Lineage between model and source code breaks on movement of source notebook. How to rectify it?

Maverick1 — Fri, 10 Sep 2021 06:05:22 GMT

If there is a registered model and it is linked with a notebook, then the lineage breaks if you move the notebook to a different path or even pull/upload a new version of the notebook.

This is not good because when someone doing its development/testing they usually do it in a messy way but if you need your code in production then it means that either

You need to move your code and then again re-train to generate the same model and then perform its movement to higher env. so that the lineage is maintained.... or
You need to keep the dev source code notebook, that too on the same path, where it had been created for dev usage.

Re: Lineage between model and source code breaks on movement of source notebook. How to rectify it?

-werners- — Fri, 10 Sep 2021 06:51:11 GMT

Hi @Saurabh Verma ,

as in your other topic, I suggest looking into MLflow as this is designed to handle all these issues.

Re: Lineage between model and source code breaks on movement of source notebook. How to rectify it?

Maverick1 — Fri, 10 Sep 2021 08:45:41 GMT

@Werner Stinckens : Hi Werners,

Thanks for the reply. But this issue is happening in MLFLow tracking itself. I wanted to know if there is a way to mitigate it or not.

Re: Lineage between model and source code breaks on movement of source notebook. How to rectify it?

-werners- — Fri, 10 Sep 2021 09:05:20 GMT

Ok I see,

do not have a clear answer for that.

But there was a session at spark summit about this (well, more CI/CD related but it might give ideas):

https://databricks.com/session_na20/productionalizing-models-through-ci-cd-design-with-mlflow

The whole ci/cd shebang might be too much for your needs but maybe you can pick some parts which are useful?

Re: Lineage between model and source code breaks on movement of source notebook. How to rectify it?

Anonymous — Fri, 10 Sep 2021 19:17:02 GMT

@Werner Stinckens - Thank you so much!

Re: Lineage between model and source code breaks on movement of source notebook. How to rectify it?

Jin1 — Sat, 11 Sep 2021 23:53:52 GMT

Hi Maverick,

I'm unable to reproduce the issue you mentioned. Where is your notebook located? Is it stored in a git-versioned Repo directory (accessed via "Repos" instead of "Workspace" icon on the navigation bar)?

Re: Lineage between model and source code breaks on movement of source notebook. How to rectify it?

Maverick1 — Sun, 12 Sep 2021 07:01:34 GMT

@Jin Zhang : For reproducing this issue:

create a notebook in your workspace account and generate a model from it.
If you go to the model stats page you can see its lineage to the original notebook.
Create a folder in your workspace and move your notebook to that folder. Now move it back to where it was before.
Go back to the model stats page and click on “source” link which represents the original notebook lineage. It will show the error as “notebook not found” although it is on the same path where it is supposed to be.

Re: Lineage between model and source code breaks on movement of source notebook. How to rectify it?

sean_owen — Wed, 15 Sep 2021 23:50:53 GMT

I also cannot reproduce this, with these exact steps (I think). After moving the notebook and moving it back, the link to it (and link to the revision) still works as expected. You are using MLflow built in to Databricks right?

Re: Lineage between model and source code breaks on movement of source notebook. How to rectify it?

Maverick1 — Mon, 27 Sep 2021 06:14:08 GMT

@Sean Owen : Yes. Managed MLFlow on databricks.

Re: Lineage between model and source code breaks on movement of source notebook. How to rectify it?

jose_gonzalez — Fri, 05 Nov 2021 16:15:50 GMT

Hi @Saurabh Verma ,

Did Sean's reply help you to solve this issue or your still are waiting for a solution to unblock you?

Re: Lineage between model and source code breaks on movement of source notebook. How to rectify it?

Maverick1 — Wed, 10 Nov 2021 09:52:16 GMT

Hi @Jose Gonzalez ,

The issue still persists. I believe this is related to the workspace version that we are using.

I have ran through the exact steps in E2 workspace version and the issue is somehow resolved there.