Databricks Community

wesleyfelipe

This article continues a technical deep dive into building large-scale Lakehouse architectures.The original platform processed billions of records across multiple markets and operated under PCI-DSS compliance requirements — a significant engineering ...

wesleyfelipe · 02-20-2026

Between 2019 and 2021, we built a large-scale lakehouse on Databricks supporting multi-market payments processing (7B+ transactions/year).If ingestion was complex (covered in Part 1), the Silver layer was even more interesting.Implementing SCD Type 1...

wesleyfelipe · 02-20-2026

Between 2019 and 2021, we built a multi-market payments data platform on Databricks that now processes more than 7 billion transactions per year across seven markets.Ingestion was by far the most operationally complex layer.To support MongoDB CDC str...

wesleyfelipe

@kiranI really enjoy reading these kinds of real-world problem cases. I like how practical and grounded your articles are. Sometimes having a solution that solves the problem now is more valuable than following the perfect best practice, especially w...

wesleyfelipe

@Hubert-Dudek Really insightful article!I’ve worked quite a lot with ADF over the past few years, but with the recent advances in orchestration in Databricks I’m starting to feel the same way — it might be time to move on from ADF. Your point about...

wesleyfelipe

@Kirankumarbs This is really great content!Streaming monitoring has always been challenging. I'm planning on writing on a similar situation I faced a few years ago too.I'm looking forward for the part 2 of your series.

wesleyfelipe

Hi @ChristianRRL !If you have your scripts in a regulat Workspace folder you might need to add the script you intend to import to the sys path: import sys sys.path.append("/Workspace/Users/your_email/my_project") But if you have your code in a Databr...

wesleyfelipe

@Saurabh2406 this is such a rich article and has so many practical takeaways! Congrats!I faced similar challenges in one of my last projects, and I could spend some time building a nice dashboard (using the system.billing tables) that helped us trac...

Databricks Community

User Stats

User Activity

Building PCI-Compliant Lakehouses: Governance Challenges Then and Modern Solutions

Scaling SCD on Databricks: Then vs Now

Is Zerobus the Future of Ingestion on Databricks? Lessons from a 7B+ Transaction Platform

Re: Streaming Failure Models: Why "It Didn't Crash" Is the Worst Outcome

Re: Move out of ADF now

Re: Streaming Failure Models: Why "It Didn't Crash" Is the Worst Outcome

Re: pytest unit testing help

Re: Designing a Cost-Efficient Databricks Lakehouse, Performance Tuning and Optimization Best Practi