What are best practices for designing a large-scale data engineering pipeline on Databricks for real

Suheb
Contributor

How do you design a scalable, reliable pipeline that handles both fast/continuous data and slower bulk data in the same system?