topic Re: Usage of Azure DevOps System.AccessToken as PAT in Databricks in Data Engineering

Usage of Azure DevOps System.AccessToken as PAT in Databricks

arsamkull — Wed, 24 Aug 2022 06:47:54 GMT

Hi there! I'm trying to use Azure DevOps Pipeline to automate Azure Databricks Repos API. Im using the following workflow:

Get an Access Token for a Databricks Service Principal using a Certificate (which works great)
Usage REST Api to generate Git Credential using System.AccessToken -> Fails with:

personal_access_token cannot be longer than 100 characters

I grabed the System.AccessToken from Azure DevOps and decoded it using JWT.io and it really is valid JWT. JWT will use a lot more than 100 chars, a resonable max char count would be 2048 (mine is 1118).

Is it possible that Databricks limits the PAT's to 100 chars? Are there alternatives for automating "git pull" using a Service Principal?

requests.post(
 
            f"{BASE_URL}/api/2.0/git-credentials",
 
            json={
 
                "git_provider": "azureDevOpsServices",
 
                "git_username": "DataHub Build Service (ORGNAME)",
 
                "personal_access_token": os.environ["SYSTEM_ACCESSTOKEN"],
 
            },
 
            headers=headers,
 
        )

Re: Usage of Azure DevOps System.AccessToken as PAT in Databricks

Soma — Tue, 01 Nov 2022 03:20:18 GMT

hi @Adrian Ehrsam

try using databricks rest api to clone the repos

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/repos

and enable file in repos to consume the data as shown in this notebook

https://docs.databricks.com/_static/notebooks/files-in-repos.html

Re: Usage of Azure DevOps System.AccessToken as PAT in Databricks

arsamkull — Tue, 01 Nov 2022 04:38:52 GMT

Well that's what I'm trying todo, but I do not get the authentication working in CI/CD because of the Length of the Access Token of Azure Devops' System.AccessToken. Must say it feels like a bug in Databricks

Re: Usage of Azure DevOps System.AccessToken as PAT in Databricks

Fual — Thu, 17 Nov 2022 13:13:37 GMT

I have the same issue, I want to automate Repos checkout during CI/CD via the rest api. Unfortunately, Azure DevOps doesn't allow Service Principals to pull from a repo and PAT validity is limited to one year max. So using the System.AccessToken would be a nice way to get around these limitations but unfortunately it's limited to 100 chars by Databricks...

Re: Usage of Azure DevOps System.AccessToken as PAT in Databricks

DBReposDev — Mon, 09 Jan 2023 08:34:40 GMT

hey everyone. We will be improving this in the coming week or two, bumping up the limit. Thank you for reporting.

Re: Usage of Azure DevOps System.AccessToken as PAT in Databricks

Srihasa_Akepati — Fri, 27 Jan 2023 08:07:40 GMT

@Adrian Ehrsam The PAT limit has been increased to 2048 now. Please check.

Re: Usage of Azure DevOps System.AccessToken as PAT in Databricks

aivashyna-vrns — Wed, 06 Aug 2025 13:49:34 GMT

Hey there, it's still not enough to use Entra ID tokens Integrating Entra ID, Azure DevOps and Databricks for Better Security in CI/CD | Databricks Blog...