Hello Databricks Community,
I'm seeking advice and insights on best practices for managing data access and permissions in Databricks. Our company currently uses a view-based approach for access control, but I'm wondering if we should transition to Databricks' native security features like row-level security (RLS) and column masks.
Our Current Approach
We create multiple view tables from upstream tables to manage permissions. For example:
- `client_info` (upstream table)
- `npii_client_info`
- `pii_client_info`
- `pii_client_info_teamA`
Concerns
1. Data lineage complexity??
2. Maintenance overhead (multiple data dictionaries)
3. Potential inconsistencies across views
4. Scalability as data and organization grow
1. What are the pros and cons of our view-based approach vs. using Databricks' RLS and column masks?
2. Has anyone successfully transitioned from a view-based system to native Databricks security features? What challenges did you face?
3. Are there specific use cases where view-based access control might be preferable?
4. How does the performance compare between these two approaches, especially for large datasets?
5. What impact does each approach have on data governance and compliance efforts?
I'd greatly appreciate any insights, experiences, or best practices you can share. Thank you in advance for your help!
