Lakebase Data API (GCP) returns jwk not found for valid service principal tokens

[eduardostzouze]

Workspace on GCP us-central1, Lakebase Autoscaling (Beta). The Data API returns 400 {"message":"jwk not found"} for every request.

Setup follows the docs: role created with databricks_create_role('<sp-uuid>', 'SERVICE_PRINCIPAL'), GRANT "<sp-uuid>" TO authenticator (pg_has_role returns true), USAGE/SELECT granted, schema exposed, schema cache refreshed, Data API disabled and re-enabled. Reproduced in a brand new project with a plain table in public.

Tokens tested: M2M workspace token (/oidc/v1/token, scope all-apis) and database credential (/api/2.0/postgres/credentials). Both have iss = workspace, aud = workspace ID, not expired, kid = _iSisQ. That kid is present in the workspace's published jwks_uri (us-central1.gcp.databricks.com/oidc/jwks.json ).

A psql/JDBC connection with the same service principal and the same database credential works fine.

Has anyone seen this on GCP? Is there a known issue with the Data API resolving JWKS on GCP workspaces?

[ivanvyd]

@eduardostzouze given your successful psql test, I'd focus next on the data api's token verification rather than changing more SQL grants. That test narrows the problem, but doesn't establish that the http endpoint can resolve the same signing key.

Could you try the same read request with a non-owner Databricks user from that workspace, with its SQL-created role granted to authenticator and equivalent read permissions? The docs explicitly exclude the database owner from Data API access. Compare the tokens' kid values too; that would help distinguish an identity-specific failure from a signing-key difference.

[eduardostzouze]

Tested with my own user token (U2M) as well — same 400 jwk not found. Since the owner restriction is applied after token validation, this shows the Data API fails signature verification for any identity in this workspace, both user (U2M) and service principal (M2M). Both token types use kid _iSisQ, which is present in the workspace's published jwks_uri (https://us-central1.gcp.databricks.com/oidc/jwks.json ). This looks like the Data API not resolving this workspace's JWKS on GCP. Can someone from the Lakebase team take a look?

[ivanvyd]

@eduardostzouze that makes an identity-specific permission issue less likely. I'd ask support to inspect the issuer/JWKS configuration actually used by the Data API verifier, rather than change more SQL grants.

To open a case, follow the Databricks GCP support instructions. With a direct Databricks support contract, go to your workspace > profile menu > Contact Support and follow the ticket-submission flow. Your email must be registered as an authorized support contact; your organization's Help Center admin can activate that access. If your support is through Google Cloud instead, use Google Cloud Support > Get help through a support case.

I'd include this thread, your clean-project reproduction, the endpoint, UTC timestamp, any returned request ID, and sanitized alg/kid/iss/aud values - not the bearer token. Ask the team to check whether the Data API verifier can retrieve and select that signing key. That should give them a concrete starting point without assuming a confirmed GCP-wide bug.

[stephen4]

Since both U2M and M2M fail with the same `kid`, that points pretty strongly to the Data API's JWKS lookup rather than permissions.

Hopefully someone from the Lakebase team can confirm whether this is a GCP-specific issue.