@dkushari Thanks for the detailed blog post. I am implementing the a similar architecture using duckdb to read from tables from data lake, and some questions.
Setup: We run a read-only external query layer, DuckDB embedded in a Go service, over Unity Catalog tables, and we have tested both the vending path your DuckDB section uses and a direct IAM alternative against replica tables. These questions come from trying to productionize exactly the read half of this post.
1. Can you confirm the data path on a SELECT? From our testing, the extension calls the temporary-table-credentials API per table, receives short-lived STS keys scoped to that table's storage prefix, and the delta extension then reads the log and parquet files directly from S3. So Databricks sits in the control path (auth, grants, credential issuance) but the bytes flow from S3 to the client with no Databricks compute involved. Is that the intended mental model?
2. Can the one-hour lifetimes be shortened? We observed both the M2M OAuth token (`expires_in: 3600`) and the vended credentials (expiration exactly one hour out) at one hour. Is either configurable downward, say to five minutes, for a tighter exposure window if something leaks? If not, is that planned?
3. What is the recommended hardening for the client id, secret, and token? A secrets manager plus rotation is the obvious baseline, and the short-lived token never needs to touch disk. But the SP's client secret is still a long-lived credential. Is there a supported path to eliminate it entirely, for example workload identity federation, so an external service exchanges its cloud identity for the Databricks token without any stored secret?
4. What other security considerations shaped the Beta? We care about two things in particular. First, `external_access_enabled` is metastore-wide, and many of us run production and non-production catalogs under one metastore, so the flag is a governance conversation; in our testing the two-key model helped (the flag alone grants nothing, and `EXTERNAL USE SCHEMA` is non-inheritable and per-schema), but is there anything further that bounds the blast radius, like restricting which principals can ever be granted it? Second, do credential vends and external commits land in `system.access.audit` in a form we can alert on, so an unexpected external reader shows up in monitoring rather than in an incident review?
