Hi @arijitsam -Great to meet you! You’ve inspired me to write an introduction post as well. I usually focus on telecom, manufacturing, and financial services, but many of the same patterns hold across industries.

My preferred approach is to ingest into Bronze using Lakeflow Connect when an appropriate connector is available. Otherwise, I’ll typically use Auto Loader for file-based ingestion, keeping transformations at that stage to a minimum.

From there, I use LDP/DLT to move data from Bronze to Silver, applying expectations for data-quality rules and quarantining records that fail validation. Depending on the source and business requirements, Silver may be an append-only streaming table or an SCD Type 2 table maintained declaratively from CDC events or source snapshots.

I also favor a metadata-driven approach to pipeline development. Configuration, column mappings, and data-quality rules live in control tables, while lightweight Python modules and notebooks read that metadata to drive pipeline behavior and orchestration dynamically. This reduces duplicated code and makes onboarding or changing a feed largely a configuration exercise rather than another custom pipeline build.

The interesting design question with B2B feeds is usually how much schema variability exists between vendors. That can quickly determine whether schema evolution is an asset or a future firedrill...