UC Delta API: managed-table READ_WRITE credentials rejected despite explicit User-Agent

[ablanchard]

Hi everyone,

I’m developing HarborSQL, an open-source SQL engine using Apache DataFusion and delta-rs. I’m working on managed Delta table writes through the UC Delta API with catalog commits.

Using the same workspace, managed table, and user identity:

- Table metadata GET succeeds (200).
- Credentials GET with operation=READ succeeds (200).
- Credentials GET with operation=READ_WRITE fails (400).

This happens with both PAT and browser-based OAuth U2M authentication.

The failing request is:

GET /api/2.1/unity-catalog/delta/v1/catalogs/{catalog}/schemas/{schema}/tables/{table}/credentials?operation=READ_WRITE
User-Agent: HarborSQL_HarborSQL/0.1.9
Accept: application/json
Content-Type: application/json

The response says:

The provided User-Agent 'HarborSQL_HarborSQL/0.1.9' is insufficient.

It also directs connector developers to contact Databricks support, but I don’t have access to support.

The header follows the documented partner User-Agent format. Using harborsql/0.1.9 produces the same error. The failure occurs during credential vending, before any data write or commit request.

Does managed-table write access require connector registration, allowlisting, or additional preview enablement? Are there client-identification requirements beyond the documented User-Agent format?

If onboarding is required for independent open-source engines implementing catalog commits, could someone point me to the appropriate process or team?

Thanks!

[GabFernandes]

Hi @ablanchard .

The behavior you're seeing is expected for the Delta protocol-level credential endpoint (/api/2.1/unity-catalog/delta/v1/.../credentials). A few things to note:

1. Managed Delta table writes from external Delta clients are in Public Preview (docs)During this preview, the Delta protocol API validates the User-Agent header for READ_WRITE operations to gate access to recognized/registered connectors. This is why READ succeeds (it's GA) while READ_WRITE fails.
2. Consider using the general credential vending API instead. The documented public endpoint is POST /api/2.1/unity-catalog/temporary-table-credentials with a JSON body containing table_id(the table's UUID) and operation: "READ_WRITE". This is the officially documented path for external engines and may not enforce the same User-Agent gating. You can get the table UUID via the ListTables API with include_manifest_capabilities enabled — look for tables marked HAS_DIRECT_EXTERNAL_ENGINE_WRITE_SUPPORT.
3. Prerequisites: the metastore must have external_access_enabled = true, and the principal needs EXTERNAL_USE_SCHEMA on the parent schema. For catalog commits specifically, the table property delta.feature.catalogManaged must be set to 'supported'.
4. For the Delta protocol path specifically, connector registration/onboarding with Databricks is likely required during the Public Preview phase. Filing a feature request via the Databricks Ideas portal or opening a GitHub issue on the unitycatalog repo would be the best way to request formal onboarding for HarborSQL.

   If my answer was helpful, please consider marking it as accepted solution!

[anuj_lathi]

Short answer: this isn't an authentication or permissions problem, so nothing on your side (PAT vs. OAuth, header formatting) will fix it. Managed-table write credential vending through the UC Delta API appears to be gated server-side by client identity, and a User-Agent that merely follows the partner format doesn't get you through.

What the evidence tells you

• Metadata GET and READ credentials both succeed with the same user and table, and both PAT and OAuth U2M fail the same way. Identity and Unity Catalog privileges are therefore not the blocker. Those are the layers that govern who can access a table [1].
• The error is specifically about the client: "The provided User-Agent … is insufficient." That is a client check, not a 403 permission denial. Changing the string to harborsql/0.1.9 returning the identical error confirms it's a recognized-client list rather than a format validator.
• It fires at READ_WRITE credential vending, before any commit call. Managed-table writes with catalog commits make Unity Catalog the commit coordinator, so Databricks controls which engines are allowed to write. Read access is the lower-risk path.

What I'd expect

I can't promise how this is handled for every account, and the behavior of this API surface isn't something Databricks guarantees to third parties today. In my experience, write access for third-party engines to catalog-managed tables is handled as a controlled enablement. You should assume it requires registration or allowlisting of the client, and possibly preview enablement on the workspace or account. There's no self-service switch I know of, and I wouldn't try to work around it by spoofing a recognized client's User-Agent. That would be fragile, and it would misidentify your engine in Databricks' telemetry.

How to get unblocked

1. Go through the account team. If your workspace belongs to an organization with a Databricks account, the account admin or the account executive/solutions architect can raise the allowlisting request internally. It's usually quicker than the support queue, and they can route it to the Unity Catalog / Delta API owners.
2. Use the partner route. An open-source engine that wants first-class integration is a technology-partner conversation. Ask your account contact to connect you with the Technology Partner Program, which is where connector registration and client identification are normally handled.
3. Make the request concrete. Include the workspace URL, the exact failing request (with the x-request-id or other request identifier from the response if present), your client name and version, the operation (READ_WRITE credential vending for catalog commits), and a link to the HarborSQL repo. That lets whoever picks it up allowlist you without a back-and-forth.
4. If you have no support entitlement, ask the owner of the workspace or account to file the ticket on your behalf. A post here can also help surface it, but it won't change the enablement on its own.

Meanwhile

You can keep developing the read path and the delta-rs commit logic against what's available to you:

• Reads via READ credential vending work today, so you can finish and test scan/planning on managed tables.
• For write-path development, use an external Delta table, where access goes through the external-location/path-credential flow rather than the managed-table catalog-commit gate. This lets you exercise the DataFusion/delta-rs write code without waiting for managed-table enablement.

Once the allowlisting is done, the same READ_WRITE call should go through with no code changes on your side.

References

[1] Authentication and access control | Databricks on AWS — https://docs.databricks.com/aws/en/security/auth