Hi @Debayan Mukherjee​, thanks for getting back to me.

Microsoft recommend not using PATs where possible as:

However, using an authentication method tied to a single person also means relying on a single point-of-failure. When a user leaves the company, the PAT driving the team application will become inaccessible to all other team members

They also say:

Additionally, PATs are bearer tokens, which can be leaked easily and fall into the wrong hands. ... we welcome you to explore service principals and managed identities instead.

Based on the risks of users leaving, and token leakage, we have a company policy which limits PAT lifetime to 90 days.

These attributes make it difficult to put a solution into production.

DevOps now supports accessing services without using a PAT, so presumably Databricks could request a bearer token for the Service Principal running the job, from Azure AD?