topic Re: Scaling Delta Sharing across Non-Databricks Recipients: Handling Credential Rotation and Lifecyc in Data Engineering

Scaling Delta Sharing across Non-Databricks Recipients: Handling Credential Rotation and Lifecycle

Khasim_1 — Sat, 19 Sep 2026 15:56:19 GMT

Hi everyone,

Hi everyone, I’m architecting a data-sharing program using Delta Sharing for external partners who do not have a Databricks workspace.

How are others automating the recipient credential lifecycle? Do you have an automated process for rotating these credentials, or is there a standard pattern for revoking access without manual intervention?
For high-scale sharing (100+ recipients), are you maintaining these via Terraform/DABs, or is there a "request-based" workflow you’ve built?
Have you encountered significant performance differences between partners using Spark vs. those using native Python/Pandas connectors?

Re: Scaling Delta Sharing across Non-Databricks Recipients: Handling Credential Rotation and Lifecyc

SumeshKashyap — Sat, 26 Sep 2026 16:01:18 GMT

The pattern that keeps this manageable at scale is treating recipients as code and running rotation as a scheduled job. Here's the breakdown:

- Set a token lifetime at the metastore level. Open-sharing bearer tokens expire according to the metastore's recipient token lifetime setting Dont use "no expiry" for external partners.

- Rotate with a grace period. The API supports this directly:

- REST: POST /api/2.1/unity-catalog/recipients/{name}/rotate-token with existing_token_expire_in_seconds

- CLI: databricks recipients rotate-token <name> <existing-token-expire-in-seconds>

- SDK: w.recipients.rotate_token(name=..., existing_token_expire_in_seconds=...)

The old token stays valid for the grace window you set (for example 7 days), so the partner can swap credentials without downtime. The response includes a new activation URL.

- Automate it as a Databricks job. Run a scheduled job that lists recipients (w.recipients.list()) and checks each token's expiration_time. For anything expiring within N days, it rotates the token and sends the new activation link to the partner. Store owner/contact/expiry metadata on the recipient itself with PROPERTIES (e.g. 'partner_contact', 'contract_end') so the job knows whom to notify.

- Deliver the activation link securely. The link is single-use: the credential file can only be downloaded once. That's good for security, but don't paste it into a normal email thread. Put it in a secure transfer channel or a portal the partner authenticates into. If a partner loses the file, rotate again rather than trying to recover it.

- For large programs, consider OIDC federation instead of bearer tokens. Delta Sharing open sharing supports OIDC token federation. Recipients authenticate with their own IdP (Entra ID, Okta, etc.), often machine-to-machine, so you don't have long-lived bearer tokens to rotate at all. It's the cleanest answer to the rotation problem at scale, as long as your partners' tooling supports it.

- Lock down and monitor.

- Attach IP access lists to each recipient where partners have stable egress IPs.

- Monitor usage from system.access.audit: filter service_name = 'unityCatalog' and action_name LIKE 'deltaSharing%' to see which recipients are actually querying. Idle recipients are candidates for offboarding.

- Make offboarding explicit. Use REVOKE SELECT ON SHARE <share> FROM RECIPIENT <recipient> to cut access to specific data, or DROP RECIPIENT to kill all of that recipient's credentials at once. Tie it to the contract end date stored in recipient properties so the same scheduled job can flag or execute it.

To keep this auditable, manage recipients, shares and grants through Terraform or Asset Bundles. Then onboarding and offboarding go through code review instead of UI clicks.

Yes when data is large (usually 50 million + rows), pandas does not perform as good as spark dataframes

Hope this helps 🙂

Re: Scaling Delta Sharing across Non-Databricks Recipients: Handling Credential Rotation and Lifecyc

Islam_hoti — Mon, 28 Sep 2026 09:27:50 GMT

Hi,

1. Credential lifecycle: If your partners have their own identity provider (Entra, Okta, etc.), consider OIDC token federation instead of bearer tokens. The partner's IdP issues short-lived tokens, so there's nothing for you to rotate or leak. It supports both user-based (U2M) and machine-to-machine (M2M) flows.

If you stay on bearer tokens:

Set a metastore-wide token lifetime so tokens expire automatically.
Automate rotation with a scheduled job that calls the rotate-token API or CLI (databricks recipients rotate-token). Set a short grace period for the old token and send the new activation link to the partner through your own channel.
Revoke access without manual steps using REVOKE SELECT ON SHARE ... FROM RECIPIENT ... or DROP RECIPIENT, triggered from your offboarding or contract-end process.
Add IP access lists on each recipient as an extra control.

2. Scale (100+ recipients): Most teams use both:

Terraform (databricks_share, databricks_recipient, databricks_grants) as the source of truth.
A request workflow in front of it, such as a ServiceNow or Jira form that opens a PR against a recipient config file (YAML/JSON). After approval, CI applies it.

This gives you an audit trail, approvals and easy offboarding. For monitoring, use system.access.audit (Delta Sharing events) to spot unused recipients and clean them up.

3. Spark vs. Python/Pandas: Yes, the difference can be large on big tables:

Spark connector: reads files in parallel, supports partition pruning and predicate pushdown, and handles CDF and streaming. Best for large or incremental reads.
Pandas connector (load_as_pandas): single-node and loads data into memory. Fine for small or medium tables, but slow or out-of-memory on large ones. Use limit and predicate hints, and newer connector versions that read with delta-kernel.

In practice: partition shared tables on common filter columns, share CDF for incremental pulls, and point pandas users to filtered views or smaller tables.