In our Dataplatform, our jobs are defined in a dataplatform_jobs.yml within a Databricks Asset Bundle, and then pushed to Databricks via an Azure Devops Pipeline (Azure Devops is where our codebase resides). Currently, this results in workflows looking like this, where they're created by the Dataplatform Service Principal, but are run as the username of a specific colleague: 

We'd like to change this, so "Run as" is also the Service Principal. This will lead to easier maintenance, and we won't have trouble if this colleague leaves the team for example. However, our workflows are connected to our Devops repo, and run on the latest version of our dev/test/acc/prd branch. As a user this runs fine, as the PAT of that specific user is used for authentication. If we change it to the sp-dataplatform, we run into authentication issues.

We could add a PAT for sp-dataplatform manually, but then this is still tied to a specific user account. This doesn't really solve the issue. 

We also tried the Azure DevOps Services (Azure Active Directory) option for Git integration within the service principal, but I believe this is only used to pull Databricks repos to Devops, instead of the other way around?

There are a lot of links and threads related to this, such as:
- https://community.databricks.com/t5/data-engineering/use-azure-service-principal-to-access-azure-dev...
- https://learn.microsoft.com/en-us/azure/databricks/repos/automate-with-ms-entra
- https://learn.microsoft.com/en-us/azure/databricks/jobs/how-to/run-jobs-with-service-principals
- https://community.databricks.com/t5/data-engineering/run-task-as-service-principal-with-code-in-azur... 

I've experimented with these options as mentioned, but I think they all serve a slightly different use case. Some colleagues who worked on different projects also didn't have a 100% satisfactory solution for this. Are we missing something; is there a way in which we can configure this to work?

Thanks in advance!