topic Re: Setup Git Integration via REST API in Data Engineering

Setup Git Integration via REST API

gbrueckl — Thu, 03 Mar 2022 15:34:22 GMT

We are currently setting up CI/CD for our Databricks workspace using Databricks Repos following the approach described in the offical docs: https://docs.databricks.com/repos.html#best-practices-for-integrating-databricks-repos-with-cicd-workflows

Obviously for our CI/CD pipelines we want to use service account(we are on Azure using AAD Service Principals) and not use regular user accounts.

Unfortunately, there is no way to configure the Git integration in a scripted way (e.g. via Repos REST API)

the docs further mention (https://docs.databricks.com/repos.html#configure-your-git-integration-with-databricks😞

For Azure DevOps, Git integration does not support Azure Active Directory tokens. You must use an Azure DevOps personal access token.

Which is another pity as ideally it would just pass the AAD token used to authenticate with the Databricks REST API further down to the Azure DevOps Repo

so how is this supposed to work as a fully automated CI/CD pipeline?

Re: Setup Git Integration via REST API

Atanu — Thu, 03 Mar 2022 17:35:45 GMT

https://docs.databricks.com/repos.html?&_ga=2.214443799.1660186726.1646314071-480220406.1638459894&_gac=1.20218826.1642457892.CjwKCAiAxJSPBhAoEiwAeO_fP8aT9M3S0jYq4SSztRUVWOPqj3LtnP8KMxOC-DirMTAKUpIJoOAC_xoC6pwQAvD_BwE#repos-api

https://docs.databricks.com/dev-tools/api/latest/repos.html#operation/get-repo

These are all currently available API regards to Repos

and below SCIM API regards to SP. But may be this wont be relevant to Repos.

https://docs.databricks.com/dev-tools/api/latest/scim/scim-sp.html#scim-api-20-serviceprincipals

Re: Setup Git Integration via REST API

gbrueckl — Thu, 03 Mar 2022 17:41:30 GMT

Thanks Atanu but none of these covers the process of setting up the connection and authentication to the git repo so I guess this is just not supported?

will this may come in the future?

Re: Setup Git Integration via REST API

Atanu — Thu, 03 Mar 2022 17:59:18 GMT

@Gerhard Brueckl could you please raise a feature request from https://docs.databricks.com/resources/ideas.html. Thank you!

Re: Setup Git Integration via REST API

gbrueckl — Fri, 04 Mar 2022 07:39:42 GMT

I am on Azure and when I go to the ideas-page of azure (https://docs.microsoft.com/en-gb/azure/databricks/resources/ideas) I get a 404 Error ...

If I open the Feedback-page directly from my DAtabricks Workspace it brings me here https://feedback.azure.com/d365community/forum/2efba7dc-ef24-ec11-b6e6-000d3a4f0da0 where most of the ideas are a couple of years old and have 0 comments (some of them have already been fixed but never got updated)

Re: Setup Git Integration via REST API

gbrueckl — Wed, 09 Mar 2022 15:52:49 GMT

again, as I am on Azure I cannot login to the ideas portal is it is solely for AWS customers

Re: Setup Git Integration via REST API

Atanu — Wed, 09 Mar 2022 16:02:16 GMT

@Gerhard Brueckl ok so for azure workspace you can share your request as per below -

The feedback option is the right place.

Thank you.

Re: Setup Git Integration via REST API

gbrueckl — Wed, 09 Mar 2022 19:29:49 GMT

Thanks, Atanu, please read my previous comments about the state of the page where this link directs you

does not look like that page got updated in the last 2 years

Re: Setup Git Integration via REST API

gbrueckl — Thu, 10 Mar 2022 08:01:36 GMT

Hi @Kaniz Fatma ,

sure, I just scrolled through all pages there and found a total of < 10 comments (non from Microsoft or Databricks) , a bunch of "ideas" that are already implemented but never got updated and a lot of ideas from Microsoft itself making it look like an orphaned backlog.

dont you share the impression that that page is dead?

A similar idea to my original question was also already submitted 2 months ago - also not comments, no nothing - still, I upvoted it but honestly I do not expect anything to happen

https://feedback.azure.com/d365community/idea/0db9b406-946b-ec11-a81b-6045bd7956bb

-gerhard

Re: Setup Git Integration via REST API

New1 — Fri, 11 Mar 2022 12:33:20 GMT

Hi, how can i trigger a job externally using Github actions?