I am trying to generate a Databricks token for a service principal (SP). I have created the SP in Azure AD and have used the Databricks rest api to add it as an admin.

When using the Databricks rest API "/api/2.0/token-management/on-behalf-of/tokens" to create a token for this SP, I get this error, {"error_code":"FEATURE_DISABLED","message":"On-behalf-of token creation for service principals is not enabled for this workspace"}.

I have enabled the personal access tokens in Admin console and I've even gone to the extent to manually add the SP to Admin permissions. I'm not sure what setting I can change in Databricks to enable this feature. Any help would be greatly appreciated.