Zerobus Ingest with Kafka compliant API and WITHOUT a callback

[DB1To3]

Is see that kafka for zerobus is still in beta.  So maybe something will change. Can someone tell me whether they are aware of an authentication handshake with this API that might AVOID the callback?

Here is the python example, which uses a sophisticated bearer-token callback: https://docs.databricks.com/aws/en/ingestion/zerobus-kafka

However, we are using a client runtime and clientside-language that is even MORE primitive than python.  Think of something like COBOL but with a tenth of the customer base.  We need to avoid the callback approach, since that is not possible in our client.  The only thing we can do is call methods like so, on a builder in preparation for the use of a producer:

CURRENT OPTIONS include setProducerOptions on builder:

 

pb:SetClientId("my-test-client").
pb:SetProducerOption("prop-name", "value").

Unfortunately the callback that provides the bearer-token is outside of our abilities.  (even if we had requested a bearer token, by way of  the endpoint "/oidc/v1/token")

Here is the description of the kafka compatible client, as-of now.  It seems to clearly indicate that auth is only happening by way of oauthbearer, and my impression is that oauthbearer is only possible via callback.  So I'm in a pretty tight spot.

The Kafka-compatible APIs run over TCP with SASL_SSL and the OAUTHBEARER mechanism and implement the producer-side subset of the Kafka protocol, which covers Produce, Metadata, ApiVersions, and the SASL handshake APIs. Consumer, admin, and transactional APIs are not available. The APIs are write-only.

Perhaps even a public / unauthenticated mechanism for using the kafka api would be ok as a temporary solution while we are doing P-o-C work ( if such a thing exists?)  We have our databricks and UC assets inside a private vnet.

Any tips would be appreciated.  Without a path forward, we will be forced to relay data from one client runtime to a different client runtime before transmitting it out to zerobus/UC.  And that is worse than the types of solutions we had twenty years ago.  I hope to avoid a regression to a more primitive solution than what we already have.

[DoTA]

Short version: as far as the current docs go, there is no non-callback path. The Zerobus Kafka endpoint accepts SASL_SSL with OAUTHBEARER only (no PLAIN, no unauthenticated mode), and the docs say the token is a Databricks OAuth token obtained from {workspace}/oidc/v1/token with authorization_details scoped to the zerobusDirectWriteApi resource and the target catalog/schema/table. Tokens last about an hour, so the docs recommend a token-provider callback rather than a static string. Since it is still Beta, I would not assume a PoC bypass exists, and I would not expect an unauthenticated option inside a private VNet either.

Two things worth checking before you build a relay:

1) "Callback" in the Python example is only the client's way of refreshing the token. Many Kafka clients can also fetch it themselves from plain properties. librdkafka-based clients have sasl.oauthbearer.method=oidc with sasl.oauthbearer.client.id, client.secret, token.endpoint.url and scope, and the Java client has sasl.oauthbearer.token.endpoint.url. If your runtime wraps one of those, SetProducerOption("sasl.oauthbearer.method", "oidc") and friends might be enough, with no code callback. Caveat I have not verified: those built-in flows send a plain client_credentials request, and Zerobus wants the authorization_details parameter, so the token may come back without the right scope. Test that first.

2) For a PoC only, you can fetch a token out of band (curl or a small scheduled job, with authorization_details), and inject it through whatever static-token option your client exposes (librdkafka has sasl.oauthbearer.config / an oauthbearer set-token call). It will expire in an hour, so it is fine for a demo and not for anything long running.

If neither works, the least bad fallback is a tiny sidecar (any language with a Kafka client that supports OAUTHBEARER) running next to your runtime and receiving records over localhost. It adds one hop, but it keeps the secret and token refresh in one place. The other option is the Zerobus gRPC/SDK ingest path instead of the Kafka API, if your runtime can speak gRPC. It is also worth asking your Databricks account team whether a simpler auth mode is planned for the Kafka API, since feedback during Beta tends to get read.

[DB1To3]

Hi @DoTA  Thanks for the reply.  It looked like AI language at first, but I see you have some good content in here!  Hopefully it is based on real world experience.  I hate it when AI answers (like the ones from Genie) send us on wild goose chases.  The problem with AI is that it never says "no you can't do X".  Instead it hallucinates five or six options (X1 to X5) and makes you chase down all of them (and waste an entire week of time) before coming to your own independent conclusion that "no you can't do X".  lol.


For your #1 option, wouldn't that approach have been mentioned in the databricks docs for python as well?  It has a similar builder.  Perhaps python doesn't use librdkafka under the hood?  Your reply gives me some hope because I was pointed to librdkafka and to the following reference, for the things I can accomplish via this new interface in my old client runtime:
https://docs.confluent.io/platform/current/clients/librdkafka/html/md_CONFIGURATION.html

... I will google for SetProducerOption("sasl.oauthbearer.method", "oidc") and try to test those approaches asap!

For #2 option I do not think that suggestion is feasible, on a technical level, since my client runtime doesn't support a callback.  That is the main problem in the first place.  I think the approach still requires the callback as the starting point.

In regards to sidecars, I'm trying to avoid another client runtime as a sidecar.  It is ugly and adds a massive amount of latency that is possibly bigger than zerobus/kafka itself.  If my original client runtime can't just do something so simple (produce messages), then it is more primitive than python and should probably be killed/abandoned.  We waited many years for kafka support in this runtime, and it will probably be 10 or 100 more years for grRPC support.  It can't do AMQP messaging either.  All it has right now is a librdkafka wrapper (without support for a callback).

[Louis_Frolio]

Hello @DB1To3, I took a look at both internal and external documentation and here is what I found.

Straight answer: there's no non-callback auth mode and no unauthenticated mode for the Zerobus Kafka API today. @DoTA has that right. The endpoint only implements SASL_SSL with OAUTHBEARER, and the token has to be a Databricks OAuth token scoped to the target table through resource and authorization_details, expiring after an hour. A private VNet doesn't change any of that, and the Kafka API doesn't support front-end Private Link yet, so you connect over the public endpoint either way.

Why the Databricks sample needs a callback: it's written with kafka-python, a pure-Python client with no built-in OAuth flow. That's a kafka-python limitation, not a Zerobus rule. Your runtime wraps librdkafka, which does have a built-in client-credentials flow, so @DoTA's first suggestion is the right one to test. Config only, no application code:

security.protocol=SASL_SSL
sasl.mechanism=OAUTHBEARER
sasl.oauthbearer.method=oidc
sasl.oauthbearer.client.id=<service principal application ID>
sasl.oauthbearer.client.secret=<service principal secret>
sasl.oauthbearer.token.endpoint.url=https://<your-workspace-url>/oidc/v1/token
sasl.oauthbearer.scope=all-apis
compression.type=none
acks=all

First check that the embedded librdkafka was built with libcurl and OpenSSL: read back builtin.features and look for oidc and http. Without them you'll get "not supported in this build" no matter what you set.

The catch, which @DoTA also called: librdkafka's OIDC flow posts grant_type and scope, and that's it. There's no documented property for resource or authorization_details, so you get a plain workspace token instead of the table-scoped one. Whether Zerobus accepts a plain token, I don't know. I'd expect SASL_AUTHENTICATION_FAILED, but the docs don't say outright. You can settle it in ten minutes without your runtime:

1. curl the token endpoint with just grant_type=client_credentials&scope=all-apis and your service principal credentials.
2. Paste that token into the Databricks Python sample as a hardcoded return from token() and produce one record.

If it works, the config above is your answer. If it fails, stock librdkafka can't mint the right token from config, and you have your "no you can't do X" with a concrete reason. (If you share results here, scrub the token and secret.)

One correction on the static-token idea: sasl.oauthbearer.config doesn't take an opaque token. It only feeds the built-in unsecured JWT generator (behind enable.sasl.oauthbearer.unsecure.jwt=true), which fabricates an unsigned test token Zerobus will reject. The only way to hand librdkafka an externally fetched token is the rd_kafka_oauthbearer_set_token() API call, so unless your wrapper exposes a "set token" method, there's no config-only injection. You'd already concluded that; this is the specific reason.

One unverified hack: I believe some librdkafka versions concatenate scope into the POST body without URL-encoding it. If yours does, a pre-encoded scope with &resource=...&authorization_details=... appended would smuggle the parameters through. One test with debug=security will tell you. It could break on the next librdkafka upgrade, so treat it as an experiment.

If both fail, the ask has two audiences. For Databricks, since the Kafka API is Beta: accept a plain client-credentials token on the Kafka endpoint, or document a config-only path. Your account team can route that. For your runtime vendor: expose oauthbearer_set_token or the refresh callback in the wrapper. Small change on their side, and it unblocks every OAUTHBEARER broker, not just Zerobus. Until then, @DoTA's localhost intermediary is the only working fallback, as unappealing as it is.

References:

• Kafka-compatible APIs with Zerobus Ingest (AWS): https://docs.databricks.com/aws/en/ingestion/zerobus-kafka
• Same page for Azure: https://learn.microsoft.com/azure/databricks/ingestion/zerobus-kafka
• Zerobus Ingest setup, service principal and token flow: https://docs.databricks.com/aws/en/ingestion/zerobus-ingest
• librdkafka configuration reference: https://docs.confluent.io/platform/current/clients/librdkafka/html/md_CONFIGURATION.html

Regards, Louis.

[DB1To3]

Hi @Louis_Frolio 
Thanks for your expertise.  I'm glad I came here first instead of reddit. 

The only reason I mentioned the vnet was in the context of NOT authenticating at all (eg. using a unauthenticated mode for producing data for zerobus ... it would be public/anonymous in the sense that anyone who can reach the endpoint can produce data for it.  But I suspect that no such thing exists, in any case.)

 >> plain workspace token instead of the table-scoped one. Whether Zerobus accepts a plain token, I don't know. 

I'll test this and get back here in the next day or two.  I have some hope that you might be on to something . Because when I looked that the "SDK" version of zerobus-ingest (for c#.Net) the auth was extremely simple (just a client ID and secret for a databricks-owned service-principal; there was no elaborate scope creation).  

Next Steps:
I'm 99% certain the runtime vendor won't work on this, if oidc/config fails.  I suspect they probably introduced the kafka client for one particular customer of theirs who paid them several million to be able to connect to kafka in a very specific set of scenarios.  I couldn't even find examples of authentication flows, so I'm guessing the other customer's scenario was unauthenticated/on-premise or something simple like that. 

So I'm assuming that I would have to eventually talk to the account team at databricks.  This kafka-compliant API is just "pretending" to be a kafka broker.  So I'm guessing that one of its primary goal is to be as versatile as possible and allow connectivity from obscure runtimes (like mine) that don't have the option to use the native SDK.  If they can find a way to allow connectivity from my unusual runtime, then that would certainly be a good measure of the versatility of this new zerobus interface.  

[Louis_Frolio · Accepted solution]

Hello @DB1To3, 

Glad we beat Reddit to it.

You read the VNet point right. There's no anonymous mode, and the Kafka endpoint is public only (no front-end Private Link yet), so "anyone who can reach it" would mean the whole internet writing into your Unity Catalog table. I wouldn't hold out for that.

On the .NET SDK, I'd set expectations before you test. The auth looks simple because the SDK does the hard part for you. You hand it a client ID, a client secret, a workspace URL, and a table name, and those four things are exactly what it needs to build the same /oidc/v1/token request the Python Kafka sample builds by hand, resource and authorization_details included, and refresh it on its own. The SDKs are open source if you want to confirm it. The Kafka docs say the token uses "the same flow as the Zerobus REST API," so the simplicity is the SDK's, not the endpoint's. There's also a July thread on the Java SDK where Lu Wang from Databricks explained that tokens minted any other way (PATs, token exchange, plain workspace tokens) fail with a 401 invalid_authorization_details. That was the gRPC path, not Kafka, so it isn't proof for your case, but I'd go in expecting the plain token to fail.

Still run it, with two guardrails so the result is clean. First, grant USE CATALOG, USE SCHEMA, SELECT, and MODIFY explicitly; ALL PRIVILEGES doesn't work for Zerobus. Second, confirm SetProducerOption actually forwards sasl.oauthbearer.* keys to librdkafka rather than silently dropping them (set one to a deliberately bad value and see whether producer creation complains). If both check out and the plain token still fails, read that as definitive: stock librdkafka config can't complete the Zerobus flow, and your wrapper doesn't expose the two things that could (token injection or a customizable token request). That's your evidence for the account team.

One idea nobody's raised yet. @DoTA mentioned the gRPC SDK path and you've ruled it out, but Zerobus also has a plain REST interface, and it may fit your runtime better than Kafka does. It's two HTTPS POSTs: one to /oidc/v1/token with the authorization_details body to mint the token, one to <zerobus-endpoint>/zerobus/v1/tables/<catalog.schema.table>/insert with a bearer header and a JSON array of records. No callback, no sidecar, no Kafka protocol. "Refresh" is just re-minting before the hour is up, on your schedule. The docs pitch REST for lightweight, chatty edge devices, and a runtime that can speak HTTPS but not gRPC is squarely in that bucket. If yours can make an HTTPS POST, I'd try this before anything else.

If you do go to the account team, I'd sharpen the ask to two options: have the Kafka endpoint accept a plain client-credentials token and enforce access from the service principal's Unity Catalog grants, or document a way to pass the table scoping through what librdkafka's OIDC flow already sends. Either one unblocks every config-only librdkafka wrapper, and librdkafka sits under the Confluent Python, Go, .NET, and Node clients too. That's a stronger pitch than "please support my runtime."

References:

• Kafka-compatible APIs with Zerobus Ingest: https://docs.databricks.com/aws/en/ingestion/zerobus-kafka
• Zerobus Ingest, REST API and token flow: https://docs.databricks.com/aws/en/ingestion/zerobus-ingest
• Zerobus SDK source: https://github.com/databricks/zerobus-sdk
• Java SDK auth thread (Lu Wang, Databricks): https://community.databricks.com/t5/administration-architecture/zerobus-ingestion-fails-in-databrick...

Regards, Louis.

[DB1To3]

Thanks for the pointer to the REST.  I can try that too.  I guess I was prejudicial against using the https port and protocol, in preference to kafka and gRPC.  It seems like it would probably take away some of the performance benefits of using streaming/zerobus, and that would end up defeating the purpose altogether.  (I have other approaches I can use to get data to a UC catalog in batch.)

But you are right that it is far better than a sidecar approach.  Thanks for mentioning it.

[DB1To3]

Hi @Louis_Frolio 
I did a lot of testing and saw the message a million times:

sasl_ssl://12345.zerobus.eastus.azuredatabricks.net:9092/bootstrap: SASL authentication error: Invalid token. (after 123ms in state AUTH_REQ)

My testing was from C#.Net which has a nuget, Confluent.Kafka.   
I'm confident that my principal was being authenticated via client secret, but the malformed token was always the problem.  As you guessed, a plain workspace token was never good enough (audience = workspace id).  You mentioned yourself that the token needs to target a table through resource and authorization_details.  But the "sasl.oathbearer.yadayadayada" don't seem to have a place to send that.  Here are the kafka docs showing the importance of those things as well.

Here is what they look like when I retrieve my own access token manually:

And here is what the final access token will look like, when they are properly specified (red)

I did lots of testing using the callback version of an access token, and kafka seems to be EXTREMELY strict with both resource and authorization_details.  If I make the smallest changes to either of them, my token will be rejected.

As I was investigating, I learned about "sasl.oauthbearer.extensions" in the librdkafka client (and Confluent.Kafka).  In theory that should be a complementary place to put additional details for sasl.oauthbearer.method=oidc.

However, in my trial-and-error I wasn't ever able to encode and relay the resource and authorization_details thru that field.  I'm not sure if they are using it yet, and I kind of wish that sasl.oauthbearer.extensions could just be hacked to specify the complete access token.  I suppose that would be some sort of serious abuse of the oidc protocol, to give it my own copy of the access token which it is supposed to be retrieving on my behalf.

I did some other things like grant all users access to all permissions in the catalog, and elevate the service principal to admin.  That didn't work either.

I will try to reach out to my account team next week and see if there is some way they can help me enable sasl.oauthbearer.method=oidc, without using a callback method.

Thanks again for all your great advice.  I think the TLDR for this conversation is that a callback is needed, even when sasl.oauthbearer.method=oidc is otherwise available in the client (for sasl.mechanism=OAUTHBEARER).  The reason is because of the strict requirements to specify the resource and authorization_details.  If I'm wrong about this, then the docs should probably share another example of authn/authz where a callback is not being used. 

[Louis_Frolio]

Glad I could be a help.

Cheers, Lou.

[Vicky_Bukta_DB]

The Zerobus team is actively working on scoped long-lived token support, which will likely help here by allowing you to avoid refreshing the token. I recommend keeping an eye out for upcoming roadmap announcements. 

[DB1To3]

Thanks for the tip.  I will definitely watch the roadmap.  
Since I'm currently waiting for the GA of Kafka, and the GA of private-link for zerobus, we can't push a "plan A" to production yet. 

And I'm not totally blocked if I want to pursue a "plan B".   @Louis_Frolio reminded me that we can just use the REST API in the short term.
Have a nice weekend.

[aayush_410]

I don’t see a way to skip the callback inside the Kafka API itself, but there are workarounds.

What the docs say (page updated Oct 1, 2026):

OAUTHBEARER is the only auth mechanism. There is no documented anonymous or unauthenticated mode, and I wouldn’t expect one, since every token is scoped to a specific table.
The token is a one-hour Databricks OAuth token, and Databricks says to supply it via a callback rather than a static string. Connections also have a bounded lifetime, and the producer reconnects and re-authenticates on its own, so a hand-pasted static token will break within the hour.
Front-end Private Link is not supported for the Kafka API. You have to use the public endpoint, which may matter given your private vnet.

Options, roughly in the order I’d try them:

• Use the Zerobus REST API instead of Kafka. The docs say the Kafka token comes from the same flow as the REST API. REST sends the token as a plain Authorization: Bearer header on each call, so no callback is needed. Your client only has to POST to /oidc/v1/token and then POST records with that header. You said you can already request the token, so this probably fits. Check whether the REST API is available to you and whether your client can do HTTPS POSTs.
• If you must stay on Kafka, use a client library that does OIDC itself. For example, librdkafka-based clients can do sasl.oauthbearer.method=oidc with a token endpoint, client id, secret and scope set purely through producer options. The catch is that Databricks needs the extra “resource” and “authorization_details” parameters in the token request, and I don’t think the built-in OIDC flow can send those. Test it, but don’t count on it.
• Put a small relay in the middle, but only for auth. Run a tiny service (an Azure Function or a container in your vnet) that holds the service principal credentials, mints the table-scoped token, and forwards to Zerobus. Your client just sends to it with a fixed key or a plain HTTPS call. This is not the “move data between two client runtimes” regression you’re worried about. It is a thin forwarder, and it also solves the private-network and public-endpoint issue.

Ask Databricks. Since it’s beta, ask your account team or SA whether a static-token or long-lived-token option or a non-callback handshake is planned. Your use case (a client that can’t run callbacks) is exactly the kind of feedback they act on.

[DB1To3]

Hi @aayush_410 

Thanks for the tips.  I'm using the REST API for now, although I'm not happy about it. (it is definitely a "plan B").

Did you see the comment from the PM (Vicky_Bukta_DB)?  They may introduce a non-callback variation of the Kafka interface that has "long-lived token support".  This may allow us to send data to kafka without the weird oauth-bearer-handshake that always involves a callback method.

>> librdkafka-based clients can do sasl.oauthbearer.method=oidc with a token endpoint, ..test it, but don’t count on it..

Yes, Louis already asked me to test and it failed.  Even for workspace admins and admins of UC.  The problem is the audience/scope (resource and authorization_details).

>>Put a small relay in the middle, but only for auth.  

How does this let me use Kafka via the standard client?  There is still no way to transmit that oauth token via librdkafka-based client, since my unusual runtime/language won't allow me to provision the callback method.

>> Ask Databricks. Since it’s beta, ask your account team or SA whether a static-token or long-lived-token option

Yes, I contacted my account team and was also able to get a reply back from Vicky_Bukta_DB (see above).  I think they are aware of the challenges that customers face when there is a missing OIDC-SASL option.

Thanks David