OAuth M2M (client-credentials) - getting error with github run

[tullius21]

I'm setting up OAuth M2M (client-credentials) authentication for a service principal, as the long-term replacement for PAT-based auth in a GitHub Actions CI/CD workflow. The token exchange itself succeeds and returns a valid, well-formed access token — but every subsequent API call made with that token, regardless of endpoint, fails with:

{"error_code": 403, "message": "should_change_password [ReqId: ...]"}

I have not been able to find this error documented anywhere for a service-principal token (service principals don't have passwords), so I'm hoping someone here has hit this before.

Steps to reproduce

1. Generate an OAuth secret for the service principal via Settings → Identity and access → Service principals → (your SP) → Secrets tab → Generate secret (scope: all APIs).
2. Exchange for a token:

curl -X POST https://<workspace-host>/oidc/v1/token \
  -u "<client_id>:<client_secret>" \
  -d "grant_type=client_credentials" \
  -d "scope=all-apis"

→ Returns a valid access_token, token_type: Bearer, expires_in: 3600. No errors at this step.

1. Use that token against any API, e.g.:

curl -X GET https://<workspace-host>/api/2.0/preview/scim/v2/Me \
  -H "Authorization: Bearer <access_token>"

→ Returns 403 should_change_password.

1. Same result reproduced independently against:
   • /api/2.0/unity-catalog/schemas?catalog_name=workspace
   • /api/2.0/clusters/list
   • /api/2.0/sql/statements (POST)

All four endpoints return the identical should_change_password message, with only the ReqId differing — this rules out an endpoint-specific permission gap and points at something account/workspace-wide.

Additional context

• Later in troubleshooting, this same should_change_password error started appearing on a personal-account PAT as well (not just the SP's OAuth token) — so whatever is triggering it does not appear to be specific to service principals or to OAuth vs. PAT auth. It seems to gate the REST/API surface broadly while the interactive workspace UI login continues to work normally with no password-change prompt.
• The service principal is confirmed Active, both in the workspace SP page and via SCIM (GET /api/2.0/preview/scim/v2/ServicePrincipals?filter=applicationId eq "<client_id>"), with entitlements including workspace-access, databricks-sql-access, workspace-consume, and group membership including admins.
• Checked Account Console → Security → Enhanced security and compliance: both "Compliance security profile for new workspaces" and "Enhanced security monitoring for new workspaces" are Disabled. These also only apply prospectively to new workspaces, so this doesn't look like the direct cause.
• Our workspace's SCIM group list includes an unusual auto-generated entry along the lines of users-clone-<date>-UTC (created by Databricks), suggesting this workspace went through some kind of clone or migration operation at some point. We suspect this may be related — possibly a stale password-policy or credential-state flag carried over from that process and incorrectly being applied to API/token-based auth generally (including to an identity type, service principals, that shouldn't have a password concept at all).
• A separate, unrelated dead/expired PAT previously existed for this SP and returned a different, expected error (401 Invalid access token) — that was resolved independently by revoking it. This should_change_password issue is distinct and newer, and only started surfacing once OAuth M2M auth was correctly configured and producing valid tokens.

Question

Has anyone seen should_change_password returned for API/token-based calls (as opposed to an actual interactive login flow)? Is this connected to a workspace clone/migration event, and if so, is there a known remediation an account admin can apply without needing a support ticket? (We don't currently have a Databricks support plan that allows filing a case directly, so posting here first.)

Happy to provide more detail — workspace region, plan tier, etc. — if useful for diagnosis.

[tullius21]

further refined this problem with below if anyone has any idea how I could address? thanks. 

• Later in troubleshooting, this same should_change_password error started appearing on a personal-account PAT as well (not just the SP's OAuth token) — so whatever is triggering it does not appear to be specific to service principals or to OAuth vs. PAT auth. It seems to gate the REST/API surface broadly while the interactive workspace UI login continues to work normally with no password-change prompt.

• The service principal is confirmed Active, both in the workspace SP page and via SCIM (GET /api/2.0/preview/scim/v2/ServicePrincipals?filter=applicationId eq "<client_id>"), with entitlements including workspace-access, databricks-sql-access, workspace-consume, and group membership including admins.

• Workspace assignment ruled out. Checked Account Console → Workspaces → [workspace] → Permissions directly: both the service principal and the admin user account are listed with Admin permission, assigned directly (not via group inheritance). So this is not an account-to-workspace assignment gap.

• Not limited to OAuth/SCIM/SQL. Also tested the On-Behalf-Of token creation endpoint (POST /api/2.0/token-management/on-behalf-of/tokens), attempting to mint a fresh SP-scoped PAT using the admin's own (already-authenticated-to-the-UI) session — this also returns the identical should_change_password, meaning even the credential-issuance path itself is blocked, not just downstream data-plane calls.

• Not a blanket account lockout. Other admin actions in the workspace UI return normal, correctly-scoped errors — e.g., attempting to manage Git credentials for a group without permission returns a proper "User does not have permission to manage Git credentials for group <id>. Please ask an admin of the group to manage the credential." This tells us the account can still receive well-formed, specific permission errors elsewhere; should_change_password appears isolated to a specific set of surfaces (SCIM, Unity Catalog, clusters, SQL Statement Execution, Token Management), which look like they may share a common underlying auth-check code path that other UI actions don't hit.

• Checked Account Console → Security → Enhanced security and compliance: both "Compliance security profile for new workspaces" and "Enhanced security monitoring for new workspaces" are Disabled. These also only apply prospectively to new workspaces, so this doesn't look like the direct cause.

• Our workspace's SCIM group list includes an unusual auto-generated entry along the lines of users-clone-<date>-UTC (created by Databricks), flagged in the UI as "Workspace local — not yet migrated to the account level," suggesting this workspace went through some kind of entitlement-control migration at some point. We tested granting this group Admin-level entitlement directly (in case group-level state was involved) — no change to the error. This rules the group out as a direct cause, though its creation timing is suspiciously close to when things were still working normally.

• A separate, unrelated dead/expired PAT previously existed for this SP and returned a different, expected error (401 Invalid access token) — that was resolved independently by revoking it. This should_change_password issue is distinct and newer, and only started surfacing once OAuth M2M auth was correctly configured and producing valid tokens.

Question

Has anyone seen should_change_password returned for API/token-based calls (as opposed to an actual interactive login flow)? Given workspace assignment, group entitlements, and security-profile settings have all been ruled out as the cause, and the failure is consistent across every credential type and issuance path we can access ourselves (PAT, OAuth M2M, OBO-token minting) — this looks like it may require a server-side flag reset on the account/identity record that only Databricks engineering can clear. Is this connected to a workspace clone/migration event, and if so, is there a known remediation an account admin can apply without needing a support ticket? (We don't currently have a Databricks support plan that allows filing a case directly, so posting here first.)

Happy to provide more detail — workspace region, plan tier, etc. — if useful for diagnosis