Databricks

arsamkull · ‎08-23-2022

Hi there! I'm trying to use Azure DevOps Pipeline to automate Azure Databricks Repos API. Im using the following workflow:

Get an Access Token for a Databricks Service Principal using a Certificate (which works great)
Usage REST Api to generate Git Credential using System.AccessToken -> Fails with:

personal_access_token cannot be longer than 100 characters

I grabed the System.AccessToken from Azure DevOps and decoded it using JWT.io and it really is valid JWT. JWT will use a lot more than 100 chars, a resonable max char count would be 2048 (mine is 1118).

Is it possible that Databricks limits the PAT's to 100 chars? Are there alternatives for automating "git pull" using a Service Principal?

requests.post(
 
            f"{BASE_URL}/api/2.0/git-credentials",
 
            json={
 
                "git_provider": "azureDevOpsServices",
 
                "git_username": "DataHub Build Service (ORGNAME)",
 
                "personal_access_token": os.environ["SYSTEM_ACCESSTOKEN"],
 
            },
 
            headers=headers,
 
        )

Soma · ‎10-31-2022

hi @Adrian Ehrsam

try using databricks rest api to clone the repos

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/repos

and enable file in repos to consume the data as shown in this notebook

https://docs.databricks.com/_static/notebooks/files-in-repos.html

arsamkull · ‎10-31-2022

Well that's what I'm trying todo, but I do not get the authentication working in CI/CD because of the Length of the Access Token of Azure Devops' System.AccessToken. Must say it feels like a bug in Databricks

Fual · ‎11-17-2022

I have the same issue, I want to automate Repos checkout during CI/CD via the rest api. Unfortunately, Azure DevOps doesn't allow Service Principals to pull from a repo and PAT validity is limited to one year max. So using the System.AccessToken would be a nice way to get around these limitations but unfortunately it's limited to 100 chars by Databricks...