- 2090 Views
- 1 replies
- 5 kudos
My experience replacing a Postgres → Kafka → DMS → S3 pipeline with Lakeflow Connect
Sharing my hands-on experience with Lakeflow Connect for anyone evaluating it for database ingestion. I recently moved data from PostgreSQL on AWS RDS into Databricks, and it replaced a painful legacy pipeline. Keeping this simple and practical.What ...
- 2090 Views
- 1 replies
- 5 kudos
- 5 kudos
Great article!
- 5 kudos
- 546 Views
- 0 replies
- 0 kudos
The Hardest Part of Our SAP Migration Wasn't the Data. It Was Timing
Part 2 of my series on building an enterprise data platform on Databricks — this one's about Silver.Part 1 covered why we ran two ingestion paths in parallel (GoldenGate CDC + JDBC batch) and kept them as separate bronze tables. If you missed it:http...
- 546 Views
- 0 replies
- 0 kudos
- 737 Views
- 0 replies
- 1 kudos
Learnings while troubleshooting Random Databricks Job Failures After Cloud Migration
One of my learning from a project.After migrating from an on-premises environment to the cloud, the data engineering team began noticing seemingly random failures in workflow-scheduled Databricks jobs.The failures appeared intermittent and often succ...
- 737 Views
- 0 replies
- 1 kudos
- 660 Views
- 0 replies
- 3 kudos
When Lakehouse Federation "loses" your tables: the silent space-in-name trap
Spent some time on a Snowflake → Databricks federation issue that didn’t throw a single error -  which is exactly what made it tricky. Sharing it in case it saves someone a debugging session.The setupConnection to Snowflake: established. Foreign cata...
- 660 Views
- 0 replies
- 3 kudos
- 943 Views
- 1 replies
- 0 kudos
From RAG Demo to Production on Databricks: 7 Things Teams Should Validate First
From RAG Demo to Production on Databricks: 7 Things Teams Should Validate FirstBy Naveen AyallaMany teams can build a RAG demo quickly.Upload documents, create embeddings, connect a model, ask a question, and show an answer.But production is differen...
- 943 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks for reading. I’m especially interested in hearing from people who have worked on real RAG or GenAI workflows.Which one has been the biggest challenge for your team?1. Choosing the right source data2. Access control and governance3. Improving r...
- 0 kudos
- 2039 Views
- 6 replies
- 11 kudos
Governance RiskOps Agent for Unity Catalog
Body:Every day, data platforms generate thousands of audit events. But here's the problem: security teams are drowning in noise.Critical risks hide in plain sight. Manual investigations take hours. Compliance gaps surface too late. And there's no int...
- 2039 Views
- 6 replies
- 11 kudos
- 11 kudos
Hi @WiliamRosa ,Good job! Really interesting use case. Thanks for sharing, and good luck in the contest!
- 11 kudos
- 1761 Views
- 1 replies
- 2 kudos
Building an Incremental Customer Data Migration Workflow in Databricks
Building an Incremental Customer Data Migration Workflow in DatabricksBy Naveen AyallaIntroductionIn many enterprise environments, customer data is spread across legacy systems that were originally designed for operational processing rather than mode...
- 1761 Views
- 1 replies
- 2 kudos
- 2 kudos
Great write-up, Naveen. Very practical and clear.I really like how you focused not just on migration, but on building a reliable incremental workflow with proper duplicate handling and governance. That’s where real value comes from.Also, connecting D...
- 2 kudos
- 2371 Views
- 0 replies
- 1 kudos
From Tableau to Databricks: Migrating KPI Dashboards with Metric Views
How to extract Tableau calculated fields, dimensions, and measures from a .twbx workbook and re-express them as a production-grade Databricks Metric View YAML — with the Sample Superstore dataset as a complete worked example, accelerated by AI coding...
- 2371 Views
- 0 replies
- 1 kudos
- 1111 Views
- 0 replies
- 0 kudos
Automating Databricks Lakeflow Connect Pipelines for CDC Databases
Hi all,Tired of paying the data movement tax or wrestling with complex manual pipeline configs?I just published a new Medium article and open-sourced a framework that fully automates Databricks Lakeflow Connect pipelines for CDC-enabled databases usi...
- 1111 Views
- 0 replies
- 0 kudos
- 741 Views
- 0 replies
- 1 kudos
Operating PostgreSQL CDC on AWS RDS with Lakeflow Connect
Over the years, I have helped organizations design and deliver large-scale data platforms, and one recurring lesson has remained constant: CDC failures are rarely caused by technology alone. They are usually the result of unclear ownership, missing o...
- 741 Views
- 0 replies
- 1 kudos
- 434 Views
- 0 replies
- 1 kudos
DAIS Community Virtual Challenge 2026: Sysl — Scanning Japanese Receipts
The ProblemLiving in Japan means getting handed receipts everywhere — convenience stores, pharmacies, restaurants. Most end up in a pocket or trash, never tracked, and the coupons go unused.The SolutionSysl is a PWA that scans any Japanese receipt au...
- 434 Views
- 0 replies
- 1 kudos
- 1139 Views
- 1 replies
- 2 kudos
DAIS Community Virtual Challenge 2026: LEGO Value Engine - using Data and AI to Find the Best LEGO
Hey everyone!For the DAIS 2026 Community Virtual Challenge, I built a LEGO Value Engine using Databricks Free Edition.This is a passion project that combined my interests of both LEGOs and Data Engineering.When a new LEGO set releases, it can be hard...
- 1139 Views
- 1 replies
- 2 kudos
- 2 kudos
GitHub Repository: Lego Value Engine Architecture Diagram (better quality version can be found here)
- 2 kudos
- 1100 Views
- 0 replies
- 3 kudos
# Retail demand forecasting on Databricks Free Edition
Hi Databricks Community,I built a retail sales forecasting system on Databricks Free Edition using the Rossmann Store Sales dataset — about 1,115 stores with daily sales over two and a half years. The goal was a 48-day forecast, the same horizon as t...
- 1100 Views
- 0 replies
- 3 kudos
- 810 Views
- 0 replies
- 2 kudos
Building a Scalable Data Pipeline with Databricks Free edition | Spark Declarative Pipelines
I recently built an end-to-end data pipeline architecture in the transportation domain, focusing on city and trip data. The pipeline follows the Bronze–Silver–Gold layered approach, where raw data is ingested into the Bronze layer, cleaned and standa...
- 810 Views
- 0 replies
- 2 kudos
- 459 Views
- 0 replies
- 0 kudos
Solving the "Untitled" Lineage Mystery in Unity Catalog
Hey everyone,Have you ever opened Databricks Catalog Explorer to audit a table, only to find the downstream job listed as "Untitled"? Databricks Unity Catalog is incredibly powerful for automated lineage, but it quietly breaks the moment you orchestr...
- 459 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 85 | |
| 76 | |
| 73 | |
| 60 | |
| 43 |
