We managed to solve this problem, however it is not an elegant solution. Databricks should simplify this.
The steps that have to be done are listed below. We are using user assigned managed identity (MI), but I assume this should work for Azure Service Principals as well.
Step 1 - Create a PAT
Create a PAT in Azure DevOps with scope Code (Read). This is done with your own account in Azure DevOps
Step 2 - Generate Databricks Entra ID access token for the MI
Note: This requires you to have attached the MI to a VM or VMSS, to connect to a bash terminal as the MI.
To get the access token, run the following command:
az account get-access-token --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d
Note: that the resource ID 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d corresponds to Azure Databricks programmatic ID [1].