https://community.databricks.com/t5/data-engineering/datamart-creation/m-p/95867#M39185 <P>In a scenario where multiple teams access overlapping but not identical datasets from a shared data lake, is it better to create separate datamarts for each team (despite data redundancy) or to maintain a single datamart and use views for team-specific access? What are the trade-offs in terms of performance, maintenance, and scalability</P> Thu, 24 Oct 2024 07:38:20 GMT billykimber 2024-10-24T07:38:20Z https://community.databricks.com/t5/data-engineering/datamart-creation/m-p/95867#M39185 <P>In a scenario where multiple teams access overlapping but not identical datasets from a shared data lake, is it better to create separate datamarts for each team (despite data redundancy) or to maintain a single datamart and use views for team-specific access? What are the trade-offs in terms of performance, maintenance, and scalability</P> Thu, 24 Oct 2024 07:38:20 GMT https://community.databricks.com/t5/data-engineering/datamart-creation/m-p/95867#M39185 billykimber 2024-10-24T07:38:20Z https://community.databricks.com/t5/data-engineering/datamart-creation/m-p/95985#M39196 <P>IMO there is no single best scenario.<BR />It depends on the case I would say.&nbsp; Both have pros and cons.<BR />If the difference between teams is really small, views could be a solution.<BR />But on the other hand, if you work on massive data, the views first have to be calculated so this can take a while.<BR />So you could use materialized views...<BR />If there is a big difference between teams, coding that in a view might not be optimal.</P><P><BR />Making separate datasets also makes sense as you can optimize each one.&nbsp; Also all logic resides in a single place (and not in view definitions).<BR />But this might be overkill for your situation.</P> Thu, 24 Oct 2024 13:55:15 GMT https://community.databricks.com/t5/data-engineering/datamart-creation/m-p/95985#M39196 -werners- 2024-10-24T13:55:15Z