Hello from a new member!

[arijitsam]

Hello Databricks Community

I finally jumped in and set up my profile this week. I wanted to drop a quick note to introduce myself and say hello to the fellow data minds here.

I've spent the last 13+ years operating as a Data Architect and Engineer, with a heavy emphasis on building scalable MarTech data products, cloud migrations, and managing compliance-heavy data streams. Currently, my engineering focus is on designing modern, B2B product data feed platforms for eCommerce Marketing.

I'm utilizing the Databricks ecosystem to scale our processing capabilities, and I'm heavily tracking features around Lakeflow, Unity Catalog, and optimization patterns using Python/PySpark for schema-heavy incoming file payloads.

I'm eager to contribute my own lessons learned in the data trenches over the years, and I'd love to know: What are your absolute go-to Databricks features or ingestion design patterns for handling this type of architecture? I would love to learn how others are optimizing these workflows.

Drop a comment, let's connect on LinkedIn, and let's build!

[davidwarner344]

Welcome to the community! Your experience will be a great addition here. Auto Loader, Delta tables, and Unity Catalog have been solid choices for scalable ingestion in my projects. Looking forward to your insights and discussions.

[cgambill]

Hi @arijitsam -Great to meet you! You’ve inspired me to write an introduction post as well. I usually focus on telecom, manufacturing, and financial services, but many of the same patterns hold across industries.

My preferred approach is to ingest into Bronze using Lakeflow Connect when an appropriate connector is available. Otherwise, I’ll typically use Auto Loader for file-based ingestion, keeping transformations at that stage to a minimum.

From there, I use LDP/DLT to move data from Bronze to Silver, applying expectations for data-quality rules and quarantining records that fail validation. Depending on the source and business requirements, Silver may be an append-only streaming table or an SCD Type 2 table maintained declaratively from CDC events or source snapshots.

I also favor a metadata-driven approach to pipeline development. Configuration, column mappings, and data-quality rules live in control tables, while lightweight Python modules and notebooks read that metadata to drive pipeline behavior and orchestration dynamically. This reduces duplicated code and makes onboarding or changing a feed largely a configuration exercise rather than another custom pipeline build.

The interesting design question with B2B feeds is usually how much schema variability exists between vendors. That can quickly determine whether schema evolution is an asset or a future firedrill...