- 1086 Views
- 2 replies
- 2 kudos
Orchestrating Irregular Databricks Jobs from external source Timestamps
Works for any event-driven workload: IoT alerts, e-commerce flash sales, financial market close processing.GoalIn this project, I needed to start Databricks jobs on an irregular basis, driven entirely by timestamps stored in PostgreSQL rather than by...
- 1086 Views
- 2 replies
- 2 kudos
- 2 kudos
@PiotrPustola -- The self-rescheduling orchestrator pattern is a really elegant solution for event-driven workloads that depend on externally managed timestamps. A few thoughts and additions that might help you and others who land on this article: AD...
- 2 kudos
- 402 Views
- 0 replies
- 3 kudos
Databricks Community Fellows February 2026 Recap - Living the Values, Rising Stars!
Databricks Community Fellows February 2026 Recap The Databricks Community Fellows are internal Brickster experts who volunteer their time to help customers succeed by answering questions in the Databricks Community forums. This month: 92 customer que...
- 402 Views
- 0 replies
- 3 kudos
- 1367 Views
- 0 replies
- 1 kudos
Building a Production‑Style SCD Type 2 Dimension on Delta Lake — Using Databricks Community Edition
If you’ve ever needed to maintain historical truth in a data warehouse, you’ve likely bumped into Slowly Changing Dimensions (SCD)—specifically Type 2. In SCD2, we keep every version of a record as it changes over time, so analysis can answer questio...
- 1367 Views
- 0 replies
- 1 kudos
- 1138 Views
- 0 replies
- 1 kudos
Databricks Metric Views - Moving Towards Business Semantics
Discussed the BI & Metrics Tax elimination using Databricks Metric Views here. Semantic Layer is a core component of the lakehouse with Metric Views. Modern stack is moving toward ai data experiences where organizations ask questions instead of build...
- 1138 Views
- 0 replies
- 1 kudos
- 3077 Views
- 3 replies
- 5 kudos
How We Built Robust Data Governance at Scale
In today's data-driven world, trust is currency—and that trust starts with quality data governed by strong principles. For one of our client, where we're on a mission to build intelligent enterprises with AI, data isn't just an asset—it's a responsib...
- 3077 Views
- 3 replies
- 5 kudos
- 330 Views
- 0 replies
- 1 kudos
Legacy BI to an Agentic Lakehouse in 90 Days -Building Autonomous AI Analytics on Databricks 2026
Why Legacy BI Is Reaching Its Limits, And What Comes NextI have always believed that the original goal of digitalization was to make data available and then find better ways to analyze it. For the past two decades, Business Intelligence has followed ...
- 330 Views
- 0 replies
- 1 kudos
- 312 Views
- 0 replies
- 1 kudos
Building PCI-Compliant Lakehouses: Governance Challenges Then and Modern Solutions
This article continues a technical deep dive into building large-scale Lakehouse architectures.The original platform processed billions of records across multiple markets and operated under PCI-DSS compliance requirements — a significant engineering ...
- 312 Views
- 0 replies
- 1 kudos
- 252 Views
- 0 replies
- 2 kudos
Databricks Lake flow - Orchestration Layer is moving to where it belongs
Discussed the BI & Metrics Tax elimination using Databricks Metric Views here. Organizations also face an older more persistent tax — the Ingestion Tax.To ingest data from a source like Salesforce or SQL Server into your Lakehouse, you typically stit...
- 252 Views
- 0 replies
- 2 kudos
- 1224 Views
- 4 replies
- 4 kudos
Resolved! Designing a Cost-Efficient Databricks Lakehouse, Performance Tuning and Optimization Best Practices
The Hidden Cost of Scaling the LakehouseOver the past few years, many organizations have successfully migrated to Databricks to modernize their data platforms. The Lakehouse architecture has enabled them to unify data engineering, analytics, and AI o...
- 1224 Views
- 4 replies
- 4 kudos
- 4 kudos
@Saurabh2406 this is such a rich article and has so many practical takeaways! Congrats!I faced similar challenges in one of my last projects, and I could spend some time building a nice dashboard (using the system.billing tables) that helped us trac...
- 4 kudos
- 448 Views
- 3 replies
- 6 kudos
🇪🇸 Por qué el DataFrame es el objeto de datos más importante en el procesamiento distribuido
En este video, creado como recordatorio para mi mala memoria a largo plazo, explico de forma sencilla: Qué es un DataFrame Cómo se distribuye en particiones Cómo se ejecuta en un cluster (driver y workers) Qué ocurre en un shuffle Relación entre...
- 448 Views
- 3 replies
- 6 kudos
- 6 kudos
Recently, I am creating some "self-reminder" videos for helping my long-term poor human memory and maybe to help others. Understand internals of Dataframes, how partitions are related to jobs, stages, shuffles and tasks and, how transformations or a...
- 6 kudos
- 254 Views
- 0 replies
- 0 kudos
🚀 LDP Tax Pipeline — Spark Declarative Pipelines on macOS (Without Databricks)
LDP Tax Pipeline — Spark Declarative Pipelines on macOS (Without Databricks)Excited to share my latest hands-on implementation of a LakeFlow Declarative Pipeline (LDP) built locally using Apache Spark 4.1 Declarative Pipelines — running entirely on ...
- 254 Views
- 0 replies
- 0 kudos
- 281 Views
- 0 replies
- 1 kudos
Scaling SCD on Databricks: Then vs Now
Between 2019 and 2021, we built a large-scale lakehouse on Databricks supporting multi-market payments processing (7B+ transactions/year).If ingestion was complex (covered in Part 1), the Silver layer was even more interesting.Implementing SCD Type 1...
- 281 Views
- 0 replies
- 1 kudos
- 345 Views
- 0 replies
- 2 kudos
Is Zerobus the Future of Ingestion on Databricks? Lessons from a 7B+ Transaction Platform
Between 2019 and 2021, we built a multi-market payments data platform on Databricks that now processes more than 7 billion transactions per year across seven markets.Ingestion was by far the most operationally complex layer.To support MongoDB CDC str...
- 345 Views
- 0 replies
- 2 kudos
- 251 Views
- 0 replies
- 0 kudos
What Championship Teams Teach Us About Modern Data Architecture.
High-performing data organizations succeed when all systems, teams, and processes are aligned toward a shared strategy. Fragmentation — separate tools for storage, governance, analytics, and AI, siloed ownership, redundant pipelines, or inconsistent ...
- 251 Views
- 0 replies
- 0 kudos
- 399 Views
- 0 replies
- 1 kudos
Lakebridge: A Developer’s Perspective on ETL Migrations
One of the recent additions to the Databricks ecosystem that caught my attention is Lakebridge, a migration accelerator aimed at legacy ETL and data warehouse workloads.Migration projects are always interesting to discuss because, in practice, they a...
- 399 Views
- 0 replies
- 1 kudos
| User | Count |
|---|---|
| 85 | |
| 72 | |
| 48 | |
| 44 | |
| 42 |
