- 12603 Views
- 3 replies
- 6 kudos
Resolved! CI/CD on Databricks with Asset Bundles (DABs) and GitHub Actions
Hi all.If you've ever manually promoted resources from dev to prod on Databricks — copying notebooks, updating configs, hoping nothing breaks — this post is for you.I've been building a CI/CD setup for a Speech-to-Text pipeline on Databricks, and I w...
- 12603 Views
- 3 replies
- 6 kudos
- 6 kudos
Hi, Great question! Databricks Asset Bundles (DABs) are the recommended approach for CI/CD on Databricks. Here is a comprehensive walkthrough. WHAT ARE DATABRICKS ASSET BUNDLES? DABs let you define your Databricks resources (jobs, pipelines, dashboar...
- 6 kudos
- 1971 Views
- 2 replies
- 2 kudos
Orchestrating Irregular Databricks Jobs from external source Timestamps
Works for any event-driven workload: IoT alerts, e-commerce flash sales, financial market close processing.GoalIn this project, I needed to start Databricks jobs on an irregular basis, driven entirely by timestamps stored in PostgreSQL rather than by...
- 1971 Views
- 2 replies
- 2 kudos
- 2 kudos
@PiotrPustola -- The self-rescheduling orchestrator pattern is a really elegant solution for event-driven workloads that depend on externally managed timestamps. A few thoughts and additions that might help you and others who land on this article: AD...
- 2 kudos
- 910 Views
- 0 replies
- 3 kudos
Databricks Community Fellows February 2026 Recap - Living the Values, Rising Stars!
Databricks Community Fellows February 2026 Recap The Databricks Community Fellows are internal Brickster experts who volunteer their time to help customers succeed by answering questions in the Databricks Community forums. This month: 92 customer que...
- 910 Views
- 0 replies
- 3 kudos
- 3708 Views
- 0 replies
- 1 kudos
Building a Production‑Style SCD Type 2 Dimension on Delta Lake — Using Databricks Community Edition
If you’ve ever needed to maintain historical truth in a data warehouse, you’ve likely bumped into Slowly Changing Dimensions (SCD)—specifically Type 2. In SCD2, we keep every version of a record as it changes over time, so analysis can answer questio...
- 3708 Views
- 0 replies
- 1 kudos
- 2949 Views
- 0 replies
- 1 kudos
Databricks Metric Views - Moving Towards Business Semantics
Discussed the BI & Metrics Tax elimination using Databricks Metric Views here. Semantic Layer is a core component of the lakehouse with Metric Views. Modern stack is moving toward ai data experiences where organizations ask questions instead of build...
- 2949 Views
- 0 replies
- 1 kudos
- 4885 Views
- 3 replies
- 5 kudos
How We Built Robust Data Governance at Scale
In today's data-driven world, trust is currency—and that trust starts with quality data governed by strong principles. For one of our client, where we're on a mission to build intelligent enterprises with AI, data isn't just an asset—it's a responsib...
- 4885 Views
- 3 replies
- 5 kudos
- 889 Views
- 0 replies
- 1 kudos
Legacy BI to an Agentic Lakehouse in 90 Days -Building Autonomous AI Analytics on Databricks 2026
Why Legacy BI Is Reaching Its Limits, And What Comes NextI have always believed that the original goal of digitalization was to make data available and then find better ways to analyze it. For the past two decades, Business Intelligence has followed ...
- 889 Views
- 0 replies
- 1 kudos
- 615 Views
- 0 replies
- 1 kudos
Building PCI-Compliant Lakehouses: Governance Challenges Then and Modern Solutions
This article continues a technical deep dive into building large-scale Lakehouse architectures.The original platform processed billions of records across multiple markets and operated under PCI-DSS compliance requirements — a significant engineering ...
- 615 Views
- 0 replies
- 1 kudos
- 2801 Views
- 4 replies
- 4 kudos
Resolved! Designing a Cost-Efficient Databricks Lakehouse, Performance Tuning and Optimization Best Practices
The Hidden Cost of Scaling the LakehouseOver the past few years, many organizations have successfully migrated to Databricks to modernize their data platforms. The Lakehouse architecture has enabled them to unify data engineering, analytics, and AI o...
- 2801 Views
- 4 replies
- 4 kudos
- 4 kudos
@Saurabh2406 this is such a rich article and has so many practical takeaways! Congrats!I faced similar challenges in one of my last projects, and I could spend some time building a nice dashboard (using the system.billing tables) that helped us trac...
- 4 kudos
- 988 Views
- 3 replies
- 6 kudos
🇪🇸 Por qué el DataFrame es el objeto de datos más importante en el procesamiento distribuido
En este video, creado como recordatorio para mi mala memoria a largo plazo, explico de forma sencilla: Qué es un DataFrame Cómo se distribuye en particiones Cómo se ejecuta en un cluster (driver y workers) Qué ocurre en un shuffle Relación entre...
- 988 Views
- 3 replies
- 6 kudos
- 6 kudos
Recently, I am creating some "self-reminder" videos for helping my long-term poor human memory and maybe to help others. Understand internals of Dataframes, how partitions are related to jobs, stages, shuffles and tasks and, how transformations or a...
- 6 kudos
- 529 Views
- 0 replies
- 0 kudos
🚀 LDP Tax Pipeline — Spark Declarative Pipelines on macOS (Without Databricks)
LDP Tax Pipeline — Spark Declarative Pipelines on macOS (Without Databricks)Excited to share my latest hands-on implementation of a LakeFlow Declarative Pipeline (LDP) built locally using Apache Spark 4.1 Declarative Pipelines — running entirely on ...
- 529 Views
- 0 replies
- 0 kudos
- 578 Views
- 0 replies
- 1 kudos
Scaling SCD on Databricks: Then vs Now
Between 2019 and 2021, we built a large-scale lakehouse on Databricks supporting multi-market payments processing (7B+ transactions/year).If ingestion was complex (covered in Part 1), the Silver layer was even more interesting.Implementing SCD Type 1...
- 578 Views
- 0 replies
- 1 kudos
- 798 Views
- 0 replies
- 2 kudos
Is Zerobus the Future of Ingestion on Databricks? Lessons from a 7B+ Transaction Platform
Between 2019 and 2021, we built a multi-market payments data platform on Databricks that now processes more than 7 billion transactions per year across seven markets.Ingestion was by far the most operationally complex layer.To support MongoDB CDC str...
- 798 Views
- 0 replies
- 2 kudos
- 438 Views
- 0 replies
- 0 kudos
What Championship Teams Teach Us About Modern Data Architecture.
High-performing data organizations succeed when all systems, teams, and processes are aligned toward a shared strategy. Fragmentation — separate tools for storage, governance, analytics, and AI, siloed ownership, redundant pipelines, or inconsistent ...
- 438 Views
- 0 replies
- 0 kudos
- 824 Views
- 0 replies
- 1 kudos
Lakebridge: A Developer’s Perspective on ETL Migrations
One of the recent additions to the Databricks ecosystem that caught my attention is Lakebridge, a migration accelerator aimed at legacy ETL and data warehouse workloads.Migration projects are always interesting to discuss because, in practice, they a...
- 824 Views
- 0 replies
- 1 kudos
| User | Count |
|---|---|
| 85 | |
| 75 | |
| 73 | |
| 60 | |
| 43 |
