Zerobus ingestion fails in Databricks

ovbioba
New Contributor

a month ago

I have premium databricks and i am trying to use ZeroBus with java SDK. 
my SP has admin rights and the righst below have been granted to the zerobus table 

GRANT USE CATALOG ON CATALOG <catalog> TO `<service-principal-id>`;
GRANT USE SCHEMA ON SCHEMA <catalog.schema> TO `<service-principal-id>`;
GRANT MODIFY, SELECT ON TABLE <catalog.schema.table> TO `<service-principal-id>`;

However when i run my application i get this error 

Caused by: com.databricks.zerobus.NonRetriableException: Specified UC token is in invalid format: Client error (401): {"error":"invalid_authorization_details","request_id":"4b188908-f314-470e-8220-ebd2dc5dfebc","error_description":"User is not authorized to the requested authorizations"}.

How can i solve this issue


0 Kudos

Lu_Wang_ENB_DBX
Databricks Employee
Databricks Employee

a month ago

Use a Databricks OAuth token for Zerobus, not a PAT or a federated/token-exchange token. The Java SDK docs now use client_id + client_secret, and Zerobus requires a token scoped to api://databricks/workspaces/<workspace_id>/zerobusDirectWriteApi with authorization_details for the table privileges.

Your specific 401: invalid_authorization_details ... User is not authorized to the requested authorizations usually means one of these:

  1. You are using the wrong token type
    Zerobus only supports the Databricks OAuth flow here; federated token exchange / workspace token paths are known to fail for Zerobus with this exact error pattern.

  2. The SDK token input is wrong for your SDK version
    Older/internal docs showed PATs, but current Java SDK usage is .oauth(clientId, clientSecret) rather than passing a PAT as uc_token.

  3. Privileges must be explicitly granted exactly as below
    ALL_PRIVILEGES does not work for Zerobus; explicit USE CATALOG, USE SCHEMA, SELECT, and MODIFY are required.

What to do

Use the Java SDK like this pattern:

ZerobusProtoStream stream = sdk.streamBuilder()
    .table("catalog.schema.table")
    .oauth(clientId, clientSecret)
    .compiledProto(descriptor)
    .build()
    .join();

or JSON similarly with .json().build(). Do not pass your SP token/PAT as uc_token unless you are on an old SDK that explicitly requires that flow.

Verify with curl first

If this fails, the issue is auth/permissions; if it succeeds, the problem is in app config.

authorization_details='[
  {"type":"unity_catalog_privileges","privileges":["USE CATALOG"],"object_type":"CATALOG","object_full_path":"<catalog>"},
  {"type":"unity_catalog_privileges","privileges":["USE SCHEMA"],"object_type":"SCHEMA","object_full_path":"<catalog>.<schema>"},
  {"type":"unity_catalog_privileges","privileges":["SELECT","MODIFY"],"object_type":"TABLE","object_full_path":"<catalog>.<schema>.<table>"}
]'

curl -X POST \
  -u "<client_id>:<client_secret>" \
  -d "grant_type=client_credentials" \
  -d "scope=all-apis" \
  -d "resource=api://databricks/workspaces/<workspace_id>/zerobusDirectWriteApi" \
  --data-urlencode "authorization_details=$authorization_details" \
  "https://<workspace-url>/oidc/v1/token"

Bottom line

Most likely fix: switch from PAT / exchanged SP token to Databricks OAuth M2M with the SP’s client_id and client_secret in the Java SDK.

View solution in original post