topic Re: I/F security about using medallion architecture in Data Engineering

I/F security about using medallion architecture

ShanQiwei — Tue, 18 Nov 2025 04:56:50 GMT

I’m new to writing requirement definitions, and I’d like to ask a question about interface (I/F) security.

My question is:
Do I need to define the authentication and security mechanisms (such as OAuth2, Managed Identity, Service Principals, etc.) between the systems shown below? Or do I also need to define security between the bronze, silver, and gold layers within the lakehouse?

Our data pipeline is:
VPC on AWS (client system) → S3 → Lakehouse (bronze → silver → gold) → Serverless compute

Re: I/F security about using medallion architecture

ManojkMohan — Tue, 18 Nov 2025 10:40:17 GMT

@ShanQiwei

Interface or Layer	Should You Define Security?	Typical Mechanisms	Reference Links
VPC → S3	Yes	IAM roles, service accounts, credentials, policies	https://docs.aws.amazon.com/AmazonS3/latest/userguide/security-best-practices.html
S3 → Lakehouse	Yes	Service principals, managed identities, access keys	https://docs.databricks.com/security/access-control/service-principals.html
Lakehouse Bronze → Silver → Gold	Sometimes (Context-Driven)	Platform roles, catalog permissions, ACLs, data masking	https://docs.databricks.com/data-governance/unity-catalog/index.html
Lakehouse → Serverless Compute	Yes	Managed identities, OAuth2, tokens, ACLs	https://learn.microsoft.com/en-us/azure/architecture/serverless/security-serverless-applications

Re: I/F security about using medallion architecture

Coffee77 — Tue, 18 Nov 2025 10:53:26 GMT

I'll try to summarize and go directly to the key points as I see this:

- Client to S3 👉 SAS Token or OAUTH 2.0 with Service to Service authentication (preferred)

- Databricks to S3 👉 Use Service Principal or Managed Identities (preferred)

- Bronze/Silver/Gold 👉 Create different catalogs per layer or different schemas/databases per catalog to place bronze, silver and gold layers. All of them under Unity Catalog governance. Then, you can set proper permissions for users, groups or service principals depending on layer they should be allowed to interact with.

- Serverless cluster 👉 You can set in "permissions" who can access and how. Establish as needed.