Part 4 of my enterprise data platform series is up - this one cover why we added a fifth layer to the standard medallion architecture.
We connected BI tools to the Gold layer and immediately hit four problems Gold alone couldn't solve:
- Schema breaks when we renamed a Gold column (three Tableau reports broke immediately)
- Three workbooks calculating vendor aging differently, none of them agreeing
- The same vendor_master join running independently across four dashboards
- Row-level filtering that we didn't want duplicated in every downstream tool
All four solutions pointed to the same thing - a Views layer between Gold and consumers.
What's in the post:
- Schema stability via views - one update instead of fixing every downstream query
- Business logic abstraction - vendor aging buckets defined once, consumed everywhere
- Row-level security with dynamic views using current_user() and Unity Catalog
- Pre-joined views for heavy consumers with Databricks Dashboard query cache
- View naming convention, Git-based version control for view definitions
- What I'd do differently - designing Views before Gold, not after
Full post on Medium: https://medium.com/@savlahanish/medallion-architecture-has-3-layers-we-built-5-heres-why-41408c71c6b...
Part 5 is where it gets messier - Tableau and Databricks Dashboards behaving differently against the same views, a decimal precision issue that cost two hours, and what happens when interactive queries and Tableau batch refreshes hit the same SQL Warehouse at 9am.
Happy to answer questions on any of the decisions - particularly around the row-level security pattern or the naming convention.
