To give you a little bit of background:

• We use terraform to deploy a resource group with multiple Azure services
• Terraform leverages an Azure Service Principal that has Owner rights to the Azure subscription
• This way, databricks is also deployed. We also define the github repo configuration within terraform, and pass in the Github username and PAT (with necessary permissions)
• A week ago, when trying to add definitions to terraform (non-databricks related), we started seeing the following error

Error: cannot read repo: PERMISSION_DENIED: Missing required permissions [View] on node with ID '1759335429158542'. Using azure-client-secret auth: host=https://adb-<XYZ>.azuredatabricks.net, azure_client_secret=***REDACTED***, 
azure_client_id=***, azure_tenant_id=***

• However, when I am unable to locate anything with that ID. I suspect it is one of the repos, but I am simply not able to find it.
• I've gone ahead and used the Git Credentials API to PATCH/modify the PAT token for our Service Principal
  • Basically logged in to Azure using the service principal, generated an access token with the scope of "2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default" << to my knowledge, this is the unique ID for databricks
  • Used CURL to GET/PATCH various API
  • Still getting the same error

Some of the commands used

#Azure login
az login --service-principal -u Env:ARM-CLIENT-ID -p Env:ARM-CLIENT-SECRET --tenant Env:ARM-TENANT-ID
 
#Generate token
AZ_TOKEN=$(az account get-access-token --scope "2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default" --tenant Env:ARM-TENANT-ID | jq -r '.accessToken')
 
#Updated the Databricks Git Credentials by using GET then PATCH
curl -X GET -H "Authorization: Bearer $DB_TOKEN" \
https://adb-7866570032917376.16.azuredatabricks.net/api/2.0/git-credentials
 
#alternatively also used
curl -X GET/PATCH -H "Authorization: Bearer $DB_TOKEN" \
-H "X-Databricks-Azure-SP-Management-Token: $AZ_TOKEN" \
-H "X-Databricks-Azure-Workspace-Resource-Id: $WS_ID" \
-d '{"personal_access_token": "$PAT", "git_username": "$GITUSER", "git_provider": "gitHub"}' \
https://$DATABRICKS_URL/api/2.0/git-credentials/
 

Any thoughts or ideas would be much appreciated.