Databricks Community

antoalphi

 Core IdeaDon’t let “1 pipeline = 1 always-on cluster” become your cost trap.Instead, design for controlled parallelism + shared compute + smart grouping.✅ A. Pipeline Sharding Strategy (Not Blind Splitting)Instead of randomly splitting 6,700 tables...

antoalphi

In Databricks Lakeflow Connect for MySQL (currently in public preview), Databricks recommends limiting each ingestion pipeline to around 250 tables, with validated testing up to 1 TB of snapshot data.However, in real-world enterprise scenarios, custo...

antoalphi

I appreciate the clear explanation and the emphasis on focusing more on business logic rather than implementation details.Have a question regarding the behavior of @DP.table.From my observations, @DP.table often results in a materialized view, even i...

antoalphi

Really enjoyed this breakdown especially how you positioned streaming tables for ingestion and materialized views for transformation. It makes designing real-time pipelines much more intuitive for SQL users.One question: in large-scale environments w...

Databricks Community

User Stats

User Activity

Solution Proposal (Cost-Optimized Architecture)

Databricks Lakeflow Connect for MySQL

Re: Spark Declarative Pipelines “How-To” Series. Part 1: How to Save Results Into A Table

Re: SDP “How-To” Series. Part 2: How to Master Streaming Tables and Materialized Views