Why Legacy BI Is Reaching Its Limits, And What Comes NextI have always believed that the original goal of digitalization was to make data available and then find better ways to analyze it. For the past two decades, Business Intelligence has followed ...
This article continues a technical deep dive into building large-scale Lakehouse architectures.The original platform processed billions of records across multiple markets and operated under PCI-DSS compliance requirements — a significant engineering ...
The Hidden Cost of Scaling the LakehouseOver the past few years, many organizations have successfully migrated to Databricks to modernize their data platforms. The Lakehouse architecture has enabled them to unify data engineering, analytics, and AI o...
@Saurabh2406 this is such a rich article and has so many practical takeaways! Congrats!I faced similar challenges in one of my last projects, and I could spend some time building a nice dashboard (using the system.billing tables) that helped us trac...
En este video, creado como recordatorio para mi mala memoria a largo plazo, explico de forma sencilla: Qué es un DataFrame Cómo se distribuye en particiones Cómo se ejecuta en un cluster (driver y workers) Qué ocurre en un shuffle Relación entre...
Recently, I am creating some "self-reminder" videos for helping my long-term poor human memory and maybe to help others. Understand internals of Dataframes, how partitions are related to jobs, stages, shuffles and tasks and, how transformations or a...
LDP Tax Pipeline — Spark Declarative Pipelines on macOS (Without Databricks)Excited to share my latest hands-on implementation of a LakeFlow Declarative Pipeline (LDP) built locally using Apache Spark 4.1 Declarative Pipelines — running entirely on ...
Between 2019 and 2021, we built a large-scale lakehouse on Databricks supporting multi-market payments processing (7B+ transactions/year).If ingestion was complex (covered in Part 1), the Silver layer was even more interesting.Implementing SCD Type 1...
Between 2019 and 2021, we built a multi-market payments data platform on Databricks that now processes more than 7 billion transactions per year across seven markets.Ingestion was by far the most operationally complex layer.To support MongoDB CDC str...
High-performing data organizations succeed when all systems, teams, and processes are aligned toward a shared strategy. Fragmentation — separate tools for storage, governance, analytics, and AI, siloed ownership, redundant pipelines, or inconsistent ...
One of the recent additions to the Databricks ecosystem that caught my attention is Lakebridge, a migration accelerator aimed at legacy ETL and data warehouse workloads.Migration projects are always interesting to discuss because, in practice, they a...
Did you know the Databricks Assistant now supports Agent Skills? If your team has common, repeatable workflows, this feature is one you need to explore. Skills provide the Databricks Assistant with a specific set of instructions to handle tasks custo...
A Key challenge for Organizations is to ensure that data metrics refer to the same for all teams. If BI logic is scattered across various tools, SQL and notebooks, a metrics tax is levied (multiple dashboards showing different revenues). Databricks M...
In financial services, data never sleeps. Trades flow in every second. Risk calculations refresh continuously. Regulatory reports demand precision. BI dashboards serve business users who expect sub-second responses. And behind all of that? A massive ...
@Nidhi_Patni Thanks for production level information!@wesleyfelipe > I have one question: what was the trade-off between using traditional partitioning with Z-ordering versus liquid clustering?Traditional partitioning with Z-ordering is kind of the o...
Hi,Is there a way to custom name an assetbundle file name and pass that to databricks bundle deploy?I mean right now I must use databricks.yml, so my question is whether there is a way to pass a custom file name.note that I don't want to embed a file...
Hi @murtadha_s The simple answer is no, but I’d like to understand the issue you’re facing so I can see if there’s anything I can help with.For example, this is how I’m using it in my production application, and it’s working quite well. We’re handlin...
When working with externally managed Delta tables and traditional partitioning strategies (for example by day, week, or month), one common challenge is:How large are my partitions actually?Before deciding whether to partition by day vs. week vs. mont...
AI didn’t fail to replace developers, It exposed something deeper.In the rush to adopt AI, many organizations assumed that coding assistants and automation would reduce engineering effort and accelerate delivery. But the reality has been different. M...
| User | Count |
|---|---|
| 85 | |
| 75 | |
| 72 | |
| 60 | |
| 43 |
