topic Re: How to compare notebooks for deployment in Data Engineering

How to compare notebooks for deployment

BeginnerBob — Tue, 20 Sep 2022 10:09:36 GMT

I have 2 Azure Databricks workspaces (test and prod)

I want to compare the notebooks to see the changes.

Are there any tools which can help do this?

Thanks,

Re: How to compare notebooks for deployment

Hubert-Dudek — Tue, 20 Sep 2022 10:33:50 GMT

Yes please use Repos and set git repositories so that you can use the standard git process during merges.

Re: How to compare notebooks for deployment

BeginnerBob — Tue, 20 Sep 2022 12:07:59 GMT

Thanks Hubert.

What's the best approach here? Do I have 2 trunks in my repos (1 for dev and the other for production)?

Can I do a compare between the 2 trunks to generate all the notebooks which need to be deployed. How do I do this compare (in DevOps, vsCode etc.)

What is the best way to do this deployment i.e. running Powershell scripts, manual overwrite etc.

Thanks,

Re: How to compare notebooks for deployment

Hubert-Dudek — Tue, 20 Sep 2022 15:29:11 GMT

I would set up two separate branches (one on each of your workspace), push from them, and then merge those branches to the "dev" or "prod" branch in the external tool (by creating a pull request). My favorite is Azure DevOps, but the most popular are GitHub and Gitlab.

Of course, you can have as many branches as you want in a separate folder in each workspace. So you can have on your workspace branch dedicated to that workspace as well "dev" and "prod", which are created after merging pull requests and are the same in both workspaces (so you don't develop on "dev" and "prod" branches, you make only pull to get them).