cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Azure Databricks Workflows with Git Integration

FedericoRaimond
New Contributor III

Hello,

I receive a very weird error when attempting to connect my workflows tasks to a remote git on azure devops.

As per documentation: "For a Git repository, the path relative to the repository root."

Then, I use directly the name of the notebook file wihtout any extension, being my notebook in the root folder. I receive the error in the first image when doing this operation.

If I follow the error message and update the path with "/" then I receive another message. (error_2)

10 REPLIES 10

szymon_dybczak
Contributor III

Hi @FedericoRaimond ,

Make sure that your notebook contains in the first line of source code following header # Databricks notebook source. If you use Scala notebook, the firs line of the source file must be // Databricks notebook source.
Use version-controlled source code in an Azure Databricks job - Azure Databricks | Microsoft Learn

I recreated this example and it worked.

Here's my devops repository with test_git.py notebook at the root level of repository:

szymon_dybczak_0-1726216675213.png

Here's workflow definition:

szymon_dybczak_1-1726216721971.png

And a successful run:

szymon_dybczak_2-1726216777217.png

 

PS. You can also try to recreate notebook, some time ago I had similar issue and notebook recreation helped me resolve this issue.




 

 

 

FedericoRaimond
New Contributor III

I had already had "# Databricks notebook source". I tried also creating a new notebook but still the error appears.

Seems like it goes well until I save the job, then it doesn't allow me to save saying a wrong error.

still error_2.PNGstill error.png

Hi @FedericoRaimond ,

One additional question, did you create test_run.py file in azure devops manually? For example, I created test_created_manually.py directly in repos and copy and paste notebook code, like below:

szymon_dybczak_0-1726221492810.png

Then I created another workflow:

szymon_dybczak_1-1726221549178.png

And when I tried to execute it I've got following error:


szymon_dybczak_2-1726221595205.png

So, what I did was to recreate this notebook using UI:

szymon_dybczak_3-1726221913306.png

Then I setup another workflow to test it:

szymon_dybczak_4-1726221957315.png

 

And now it works:

szymon_dybczak_5-1726222002307.png

 

So, try once again recreate your notebook using databricks UI and then copy and paste each cell and try again to recreate workflow from scratch.
It's weird, I suppose that there could be some hidden characters/encoding issue within your notebook that's causign this problem.




 

I actually created it from UI in repos and committed and pushed to Azure DevOps ๐Ÿ˜….

I also believe the error you get in the manual attempt is even different from mine. On my side, after I set correctly the connection to the git branch, it doesn't even allow me to save the job edits. This isn't just an error I get with notebooks but I believe the issue is somwhere in the integration with Azure DevOps...

Haha, could be. Pretty buggy experience. To be honest, I'm running out of ideas. Last one idea, maybe your personal access token has insufficient permission?

I have a Full Read access token, even created a new one to test again. Still same issue.

Can you try a Github token with read and write access? The write access is required at the moment. Our engineering team is working with Github to understand why the write access is needed and see whether we can remove this requirement. 

Hello,

As described above, I'm using Azure DevOps. I used a Full Access Token, so it has both read and write access.

nicole_lu_PM
Contributor III

Hi Federico,

The error in Error 1.png didn't look right. Since you already selected the git source for the job, you should be able to use a relative path. If you continue to run into this issue, can you please submit a support ticket if you have a Support subscription?

Thank you,

Nicole

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group