- 279 Views
- 1 replies
- 2 kudos
Building an Incremental Customer Data Migration Workflow in Databricks
Building an Incremental Customer Data Migration Workflow in DatabricksBy Naveen AyallaIntroductionIn many enterprise environments, customer data is spread across legacy systems that were originally designed for operational processing rather than mode...
- 279 Views
- 1 replies
- 2 kudos
- 2 kudos
Great write-up, Naveen. Very practical and clear.I really like how you focused not just on migration, but on building a reliable incremental workflow with proper duplicate handling and governance. That’s where real value comes from.Also, connecting D...
- 2 kudos
- 254 Views
- 0 replies
- 1 kudos
From Tableau to Databricks: Migrating KPI Dashboards with Metric Views
How to extract Tableau calculated fields, dimensions, and measures from a .twbx workbook and re-express them as a production-grade Databricks Metric View YAML — with the Sample Superstore dataset as a complete worked example, accelerated by AI coding...
- 254 Views
- 0 replies
- 1 kudos
- 146 Views
- 0 replies
- 0 kudos
Automating Databricks Lakeflow Connect Pipelines for CDC Databases
Hi all,Tired of paying the data movement tax or wrestling with complex manual pipeline configs?I just published a new Medium article and open-sourced a framework that fully automates Databricks Lakeflow Connect pipelines for CDC-enabled databases usi...
- 146 Views
- 0 replies
- 0 kudos
- 156 Views
- 0 replies
- 1 kudos
Operating PostgreSQL CDC on AWS RDS with Lakeflow Connect
Over the years, I have helped organizations design and deliver large-scale data platforms, and one recurring lesson has remained constant: CDC failures are rarely caused by technology alone. They are usually the result of unclear ownership, missing o...
- 156 Views
- 0 replies
- 1 kudos
- 150 Views
- 0 replies
- 1 kudos
DAIS Community Virtual Challenge 2026: Sysl — Scanning Japanese Receipts
The ProblemLiving in Japan means getting handed receipts everywhere — convenience stores, pharmacies, restaurants. Most end up in a pocket or trash, never tracked, and the coupons go unused.The SolutionSysl is a PWA that scans any Japanese receipt au...
- 150 Views
- 0 replies
- 1 kudos
- 358 Views
- 1 replies
- 2 kudos
DAIS Community Virtual Challenge 2026: LEGO Value Engine - using Data and AI to Find the Best LEGO
Hey everyone!For the DAIS 2026 Community Virtual Challenge, I built a LEGO Value Engine using Databricks Free Edition.This is a passion project that combined my interests of both LEGOs and Data Engineering.When a new LEGO set releases, it can be hard...
- 358 Views
- 1 replies
- 2 kudos
- 2 kudos
GitHub Repository: Lego Value Engine Architecture Diagram (better quality version can be found here)
- 2 kudos
- 138 Views
- 0 replies
- 1 kudos
# Retail demand forecasting on Databricks Free Edition
Hi Databricks Community,I built a retail sales forecasting system on Databricks Free Edition using the Rossmann Store Sales dataset — about 1,115 stores with daily sales over two and a half years. The goal was a 48-day forecast, the same horizon as t...
- 138 Views
- 0 replies
- 1 kudos
- 202 Views
- 0 replies
- 2 kudos
Building a Scalable Data Pipeline with Databricks Free edition | Spark Declarative Pipelines
I recently built an end-to-end data pipeline architecture in the transportation domain, focusing on city and trip data. The pipeline follows the Bronze–Silver–Gold layered approach, where raw data is ingested into the Bronze layer, cleaned and standa...
- 202 Views
- 0 replies
- 2 kudos
- 138 Views
- 0 replies
- 0 kudos
Solving the "Untitled" Lineage Mystery in Unity Catalog
Hey everyone,Have you ever opened Databricks Catalog Explorer to audit a table, only to find the downstream job listed as "Untitled"? Databricks Unity Catalog is incredibly powerful for automated lineage, but it quietly breaks the moment you orchestr...
- 138 Views
- 0 replies
- 0 kudos
- 219 Views
- 0 replies
- 0 kudos
Why Your Delta MERGE is 5x Slower Than an Overwrite (And How to Fix It)
Hey everyone,We’ve all been there: a Delta Lake MERGE job that should take 20 minutes drags on for 90 minutes, while a full overwrite of the same table finishes in under 20. When an overwrite outpaces a selective merge, it's a massive red flag that y...
- 219 Views
- 0 replies
- 0 kudos
- 337 Views
- 2 replies
- 4 kudos
How I Passed the Databricks GenAI Engineer Associate — A No-Fluff Study Guide
Hello Everyone, As a Data & Analytics Engineer with experience spanning ETL, data engineering, solution design, and data platform engineering, I currently work Azure Data Ecosystem involving Azure Databricks, Terraform, and CI/CD pipelines — building...
- 337 Views
- 2 replies
- 4 kudos
- 161 Views
- 0 replies
- 0 kudos
Converting stored procedures to PySpark
Hi Everyone,I just publsihed an detailed article in medium for migrating the stored procedures to pyspark #https://medium.com/p/909c5c700ffd?postPublishedType=initial
- 161 Views
- 0 replies
- 0 kudos
- 175 Views
- 0 replies
- 1 kudos
Why We Used Two Bronze Tables Instead of One — And Why It Mattered
Part 1 of a 5-part series on building an enterprise data platform on Databricks.When migrating a large retail conglomerate's SAP HANA platform to Databricks, we needed both historicalcompleteness and near-real-time freshness from day one.That require...
- 175 Views
- 0 replies
- 1 kudos
- 299 Views
- 0 replies
- 0 kudos
Stop giving your Databricks AI Agents 50 tools to manage
If you are building enterprise Generative AI, you know the pain of the "Monolithic Agent Bottleneck": passing dozens of tools to a single AI supervisor leads to hallucinated routing, massive context bloat, and security nightmares.With the recent May ...
- 299 Views
- 0 replies
- 0 kudos
- 228 Views
- 0 replies
- 1 kudos
Power BI - DBX Unity catalog Lineage Syncer
Hi folks, So I have recently, as a side project been working on a tool, Link to my github: https://github.com/JeffDenzel/lineage-syncer.The problemThe reason for it is that DataBricks allows you to put external data assets into your UC. However, ther...
- 228 Views
- 0 replies
- 1 kudos
| User | Count |
|---|---|
| 85 | |
| 74 | |
| 57 | |
| 44 | |
| 44 |
