topic Re: Call Databricks notebook in a specific branch from Azure Data Factory? in Data Engineering

Call Databricks notebook in a specific branch from Azure Data Factory?

baatchus — Mon, 27 Sep 2021 14:04:08 GMT

I'm using the new Databricks Repos functionality and in Azure Data Factory UI for the notebook activity you can browse the Databricks workspace and select Repos > username > project > folder > notebook.

Is it possible to call a Databricks notebook in a specific branch from Data Factory?

Re: Call Databricks notebook in a specific branch from Azure Data Factory?

Ryan_Chynoweth — Mon, 27 Sep 2021 18:43:39 GMT

I believe that the branch is defaulted to the branch that is set via the UI or by the repos REST API.

In your workflow I would think that you will need to call a REST api to change the branch of the repo, Then you would execute the notebook. For example, this may be a situation where you create a release branch and want to dynamically change the code that is executing. You may need to make a pull request as well but not certain.

Let me know if this helps. I can provide more detail.

Re: Call Databricks notebook in a specific branch from Azure Data Factory?

-werners- — Tue, 28 Sep 2021 09:08:11 GMT

Data Factory will take the branch which was set by the Databricks Web UI.

So you can go using the REST API, but what we do is just use different folders.

You can create folders in the REPOS and under each folder you can select another branch.

(f.e. your own user folder for dev, and another one for production/QA etc)

Re: Call Databricks notebook in a specific branch from Azure Data Factory?

baatchus — Tue, 28 Sep 2021 11:59:30 GMT

Ok so you create top level folders called Production and then sync the repo and have it at the main/master branch.

Do you call the REST API from Data Factory to update the branch before running the notebook activity?

Do you parameterize the notebook path in Data Factory for the top level folder in Databricks Repos?

How does a typical workflow look like? Have you configured Databricks Repos with top level folders on all environments? Dev, Stage, Production?

So many questions but so little documentation and guidance from Databricks/Microsoft on this part..

Re: Call Databricks notebook in a specific branch from Azure Data Factory?

-werners- — Tue, 28 Sep 2021 12:10:37 GMT

Exactly.

We have a folder called 'Production' which is set to main, always.

The only time we do something in this folder is to pull the (approved) changes from git.

Data Factory uses this path for executing notebooks.

You can do a rest call to make sure the branch is set to main, but we do not do that as we are disciplined enoug to not mess with it 🙂

And this you can do also for QA (if this resides in the same databricks workspace)

Further, each developer has his own folder. In this folder he/she can create branches and do commits. You can only have one branch with open changes, so if you want to switch branch you first have to commit.

Why not a single 'Development' folder instead of one per developer? Simple: each folder can only have one branch active. If you work with multiple people into one folder, big mess!

If you have multiple workspace, the same principles are valid. A folder per developer and one 'golden' branch which gets promoted to other workspaces.

Re: Call Databricks notebook in a specific branch from Azure Data Factory?

Maksym — Tue, 20 Dec 2022 14:58:19 GMT

Greetings, I have similar problem.

Did you try to use Databricks workflows instead and schedule them instead on Data Factory?

Because inside workflows it is possible to select a specific branch, so it may actually work.

What do you think?