topic Re: Azure Databricks Workflows with Git Integration in Data Engineering

Azure Databricks Workflows with Git Integration

FedericoRaimond — Thu, 12 Sep 2024 13:56:46 GMT

Hello,

I receive a very weird error when attempting to connect my workflows tasks to a remote git on azure devops.

As per documentation: "For a Git repository, the path relative to the repository root."

Then, I use directly the name of the notebook file wihtout any extension, being my notebook in the root folder. I receive the error in the first image when doing this operation.

If I follow the error message and update the path with "/" then I receive another message. (error_2)

Re: Azure Databricks Workflows with Git Integration

szymon_dybczak — Fri, 13 Sep 2024 08:42:13 GMT

Hi @FedericoRaimond ,

Make sure that your notebook contains in the first line of source code following header # Databricks notebook source. If you use Scala notebook, the firs line of the source file must be // Databricks notebook source.
Use version-controlled source code in an Azure Databricks job - Azure Databricks | Microsoft Learn

I recreated this example and it worked.

Here's my devops repository with test_git.py notebook at the root level of repository:

Here's workflow definition:

And a successful run:

PS. You can also try to recreate notebook, some time ago I had similar issue and notebook recreation helped me resolve this issue.

Re: Azure Databricks Workflows with Git Integration

FedericoRaimond — Fri, 13 Sep 2024 08:57:22 GMT

I had already had "# Databricks notebook source". I tried also creating a new notebook but still the error appears.

Seems like it goes well until I save the job, then it doesn't allow me to save saying a wrong error.

Re: Azure Databricks Workflows with Git Integration

szymon_dybczak — Fri, 13 Sep 2024 10:08:06 GMT

Hi @FedericoRaimond ,

One additional question, did you create test_run.py file in azure devops manually? For example, I created test_created_manually.py directly in repos and copy and paste notebook code, like below:

Then I created another workflow:

And when I tried to execute it I've got following error:

So, what I did was to recreate this notebook using UI:

Then I setup another workflow to test it:

And now it works:

So, try once again recreate your notebook using databricks UI and then copy and paste each cell and try again to recreate workflow from scratch.
It's weird, I suppose that there could be some hidden characters/encoding issue within your notebook that's causign this problem.

Re: Azure Databricks Workflows with Git Integration

FedericoRaimond — Fri, 13 Sep 2024 11:34:32 GMT

I actually created it from UI in repos and committed and pushed to Azure DevOps 😅.

I also believe the error you get in the manual attempt is even different from mine. On my side, after I set correctly the connection to the git branch, it doesn't even allow me to save the job edits. This isn't just an error I get with notebooks but I believe the issue is somwhere in the integration with Azure DevOps...

Re: Azure Databricks Workflows with Git Integration

szymon_dybczak — Fri, 13 Sep 2024 11:40:20 GMT

Haha, could be. Pretty buggy experience. To be honest, I'm running out of ideas. Last one idea, maybe your personal access token has insufficient permission?

Re: Azure Databricks Workflows with Git Integration

FedericoRaimond — Fri, 13 Sep 2024 11:53:48 GMT

I have a Full Read access token, even created a new one to test again. Still same issue.

Re: Azure Databricks Workflows with Git Integration

nicole_lu_PM — Fri, 27 Sep 2024 18:48:50 GMT

Can you try a Github token with read and write access? The write access is required at the moment. Our engineering team is working with Github to understand why the write access is needed and see whether we can remove this requirement.

Re: Azure Databricks Workflows with Git Integration

FedericoRaimond — Tue, 01 Oct 2024 07:10:42 GMT

Hello,

As described above, I'm using Azure DevOps. I used a Full Access Token, so it has both read and write access.

Re: Azure Databricks Workflows with Git Integration

nicole_lu_PM — Tue, 01 Oct 2024 21:36:17 GMT

Hi Federico,

The error in Error 1.png didn't look right. Since you already selected the git source for the job, you should be able to use a relative path. If you continue to run into this issue, can you please submit a support ticket if you have a Support subscription?

Thank you,

Nicole