just to add a context for probably the underlying issue requiring an account level service principal (with OAuth).

I experienced the same issue while using username and password as in the case how TF provider was configured for existing workspaces created prior to 11/2023.  Looks like Databricks is expecting/enforcing an account level service principal for TF provider for new workspaces after 11/23.