topic Re: What are best practices for designing a large-scale data engineering pipeline on Databricks for in Data Engineering https://community.databricks.com/t5/data-engineering/what-are-best-practices-for-designing-a-large-scale-data/m-p/139275#M51133 <P>Very generic question <span class="lia-unicode-emoji" title=":slightly_smiling_face:">🙂</span> Here are general rules and best practices related to Databricks well-architected framework:&nbsp;<A href="https://docs.databricks.com/aws/en/lakehouse-architecture/well-architected" target="_blank">https://docs.databricks.com/aws/en/lakehouse-architecture/well-architected</A>&nbsp;Take a deeper look on operational excellence, reliability and performance efficiency. On the other hand, try to adopt a mediallion architecture to logically organize data&nbsp;<A href="https://www.databricks.com/glossary/medallion-architecture" target="_blank">https://www.databricks.com/glossary/medallion-architecture</A>&nbsp;and usage of Unity catalog to centrally control and governance data.</P> Mon, 17 Nov 2025 07:52:01 GMT Coffee77 2025-11-17T07:52:01Z What are best practices for designing a large-scale data engineering pipeline on Databricks for real https://community.databricks.com/t5/data-engineering/what-are-best-practices-for-designing-a-large-scale-data/m-p/139265#M51128 <P>How do you design a scalable, reliable pipeline that handles both fast/continuous data and slower bulk data in the same system?</P> Mon, 17 Nov 2025 06:39:52 GMT https://community.databricks.com/t5/data-engineering/what-are-best-practices-for-designing-a-large-scale-data/m-p/139265#M51128 Suheb 2025-11-17T06:39:52Z Re: What are best practices for designing a large-scale data engineering pipeline on Databricks for https://community.databricks.com/t5/data-engineering/what-are-best-practices-for-designing-a-large-scale-data/m-p/139275#M51133 <P>Very generic question <span class="lia-unicode-emoji" title=":slightly_smiling_face:">🙂</span> Here are general rules and best practices related to Databricks well-architected framework:&nbsp;<A href="https://docs.databricks.com/aws/en/lakehouse-architecture/well-architected" target="_blank">https://docs.databricks.com/aws/en/lakehouse-architecture/well-architected</A>&nbsp;Take a deeper look on operational excellence, reliability and performance efficiency. On the other hand, try to adopt a mediallion architecture to logically organize data&nbsp;<A href="https://www.databricks.com/glossary/medallion-architecture" target="_blank">https://www.databricks.com/glossary/medallion-architecture</A>&nbsp;and usage of Unity catalog to centrally control and governance data.</P> Mon, 17 Nov 2025 07:52:01 GMT https://community.databricks.com/t5/data-engineering/what-are-best-practices-for-designing-a-large-scale-data/m-p/139275#M51133 Coffee77 2025-11-17T07:52:01Z