https://community.databricks.com/t5/data-engineering/questions-on-bronze-silver-gold-data-set-layering/m-p/25892#M18068 <P>I have a DB-savvy customer who is concerned their silver/gold layer is becoming too expensive.&nbsp; These layers are heavily denormalized, focused on logical business entities (customers, claims, services, etc), and maintained by MERGEs.&nbsp; They cannot predict which rows / columns are going to be updated so ZORDER and partitioning have proved ineffective at limiting files to change and thus compute cost.&nbsp; They want to understand EDW patterns used by other large, sophisticated data engineering teams. </P> Mon, 07 Jun 2021 18:08:45 GMT RonanStokes_DB 2021-06-07T18:08:45Z https://community.databricks.com/t5/data-engineering/questions-on-bronze-silver-gold-data-set-layering/m-p/25892#M18068 <P>I have a DB-savvy customer who is concerned their silver/gold layer is becoming too expensive.&nbsp; These layers are heavily denormalized, focused on logical business entities (customers, claims, services, etc), and maintained by MERGEs.&nbsp; They cannot predict which rows / columns are going to be updated so ZORDER and partitioning have proved ineffective at limiting files to change and thus compute cost.&nbsp; They want to understand EDW patterns used by other large, sophisticated data engineering teams. </P> Mon, 07 Jun 2021 18:08:45 GMT https://community.databricks.com/t5/data-engineering/questions-on-bronze-silver-gold-data-set-layering/m-p/25892#M18068 RonanStokes_DB 2021-06-07T18:08:45Z