Zerobus ingestion fails in Databricks Free Edition using Service Principal OAuth

[yyonm]

Hi everyone!.

I developed a Python producer that sends asynchronous dummy messages to Zerobus. I tested it in the Databricks workspace I use at work, and everything worked as expected. The messages were successfully ingested, and I was able to verify that the data was written to Unity Catalog.

However, when I try to send the same messages to Databricks Free Edition, I encounter the following error:

rust_stream = await self._inner.create_stream(client_id, client_secret, table_properties, options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_zerobus_core.ZerobusException: Specified UC token is in invalid format: Client error (401): {"error":"invalid_authorization_details","request_id":"5a306772-908c-4e2c-af71-99c532274b7d","error_description":"User is not authorized to the requested authorizations"}.

Additional information from the Free Edition workspace:

- Authentication is performed using a Service Principal with an OAuth client secret.
- The catalog, schema, and table have already been created in Unity Catalog.
- The Service Principal has ALL PRIVILEGES on the catalog, schema, and table.
- The exact same code works correctly in a non-Free Databricks workspace, so the issue appears to be specific to Databricks Free Edition.

Has anyone experienced this issue?, or does anyone know whether Zerobus has any limitations or requires additional configuration in Databricks Free Edition?

Thank you in advance for your time and any guidance or suggestions you can provide.

[balajij8]

Free Edition has several administrative restrictions as it's not for commercial use.

• Authentication is limited to Email OTP, Google/Microsoft Sign
• No access to the account console or account-level APIs
• No SSO or SCIM support

Service Principals with OAuth client secrets require account level infrastructure that is not available in Free Edition leading to the zerobus authentication issues.

[iyashk-DB · Accepted solution]

Hi, it's worth verifying the exact Unity Catalog permissions before concluding that this is purely an edition limitation.

Zerobus specifically requires the following grants:

GRANT USE CATALOG ON CATALOG <catalog> TO `<service-principal-id>`;
GRANT USE SCHEMA ON SCHEMA <catalog.schema> TO `<service-principal-id>`;
GRANT MODIFY, SELECT ON TABLE <catalog.schema.table> TO `<service-principal-id>`;

If those permissions are already covered by an ALL PRIVILEGES grant, then the other reply is probably pointing in the right direction.

Databricks Free Edition does not provide access to the account console or account-level APIs, and service principal OAuth authentication depends on that account-level identity infrastructure. A 401 returned during token authorization—before Unity Catalog has an opportunity to evaluate table permissions—fits that explanation more closely than a simple Unity Catalog permissions issue.

That said, I couldn't find documentation explicitly stating that Zerobus is unsupported on Free Edition. Rather than continuing to troubleshoot, I'd recommend confirming with Databricks Support whether service principal authentication for Zerobus is expected to work on Free Edition at all.

[yyonm]

You're absolutely right. The grants must be exactly as specified in the documentation; having the ALL PRIVILEGES permission does not cover the other required grants. After executing those three GRANT statements on the catalog, schema, and table, the error message changed to:

message: "Unsupported table kind. Tables created in default storage are not supported. Error Code: 4024, Error State: 0."

This happens because I'm using a MANAGED table with the default storage location. As I understand it, I now need to create a new table that uses a specific storage location, for example in an AWS S3 bucket, instead of the default storage, as indicated in the documentation:

"The connector supports writing only to managed Delta tables. Writing to default storage is not supported."

Source: https://docs.databricks.com/aws/en/ingestion/zerobus-limits

Thank you so much for your help!