Azure Databricks Workflows with Git Integration

[FedericoRaimond]

Hello,

I receive a very weird error when attempting to connect my workflows tasks to a remote git on azure devops.

As per documentation: "For a Git repository, the path relative to the repository root."

Then, I use directly the name of the notebook file wihtout any extension, being my notebook in the root folder. I receive the error in the first image when doing this operation.

If I follow the error message and update the path with "/" then I receive another message. (error_2)

[szymon_dybczak]

Hi @FedericoRaimond ,

Make sure that your notebook contains in the first line of source code following header # Databricks notebook source. If you use Scala notebook, the firs line of the source file must be // Databricks notebook source.
Use version-controlled source code in an Azure Databricks job - Azure Databricks | Microsoft Learn

I recreated this example and it worked.

Here's my devops repository with test_git.py notebook at the root level of repository:

Here's workflow definition:

And a successful run:

PS. You can also try to recreate notebook, some time ago I had similar issue and notebook recreation helped me resolve this issue.

[FedericoRaimond]

I had already had "# Databricks notebook source". I tried also creating a new notebook but still the error appears.

Seems like it goes well until I save the job, then it doesn't allow me to save saying a wrong error.

[szymon_dybczak]

Hi @FedericoRaimond ,

One additional question, did you create test_run.py file in azure devops manually? For example, I created test_created_manually.py directly in repos and copy and paste notebook code, like below:

Then I created another workflow:

And when I tried to execute it I've got following error:

So, what I did was to recreate this notebook using UI:

Then I setup another workflow to test it:

And now it works:

So, try once again recreate your notebook using databricks UI and then copy and paste each cell and try again to recreate workflow from scratch.
It's weird, I suppose that there could be some hidden characters/encoding issue within your notebook that's causign this problem.

[FedericoRaimond]

I actually created it from UI in repos and committed and pushed to Azure DevOps 😅.

I also believe the error you get in the manual attempt is even different from mine. On my side, after I set correctly the connection to the git branch, it doesn't even allow me to save the job edits. This isn't just an error I get with notebooks but I believe the issue is somwhere in the integration with Azure DevOps...

[szymon_dybczak]

Haha, could be. Pretty buggy experience. To be honest, I'm running out of ideas. Last one idea, maybe your personal access token has insufficient permission?

[FedericoRaimond]

I have a Full Read access token, even created a new one to test again. Still same issue.

[nicole_lu_PM]

Can you try a Github token with read and write access? The write access is required at the moment. Our engineering team is working with Github to understand why the write access is needed and see whether we can remove this requirement. 

[FedericoRaimond]

Hello,

As described above, I'm using Azure DevOps. I used a Full Access Token, so it has both read and write access.

[nicole_lu_PM]

Hi Federico,

The error in Error 1.png didn't look right. Since you already selected the git source for the job, you should be able to use a relative path. If you continue to run into this issue, can you please submit a support ticket if you have a Support subscription?

Thank you,

Nicole