- 999 Views
- 6 replies
- 11 kudos
Governance RiskOps Agent for Unity Catalog
Body:Every day, data platforms generate thousands of audit events. But here's the problem: security teams are drowning in noise.Critical risks hide in plain sight. Manual investigations take hours. Compliance gaps surface too late. And there's no int...
- 999 Views
- 6 replies
- 11 kudos
- 11 kudos
Hi @WiliamRosa ,Good job! Really interesting use case. Thanks for sharing, and good luck in the contest!
- 11 kudos
- 965 Views
- 1 replies
- 2 kudos
Building an Incremental Customer Data Migration Workflow in Databricks
Building an Incremental Customer Data Migration Workflow in DatabricksBy Naveen AyallaIntroductionIn many enterprise environments, customer data is spread across legacy systems that were originally designed for operational processing rather than mode...
- 965 Views
- 1 replies
- 2 kudos
- 2 kudos
Great write-up, Naveen. Very practical and clear.I really like how you focused not just on migration, but on building a reliable incremental workflow with proper duplicate handling and governance. That’s where real value comes from.Also, connecting D...
- 2 kudos
- 810 Views
- 0 replies
- 1 kudos
From Tableau to Databricks: Migrating KPI Dashboards with Metric Views
How to extract Tableau calculated fields, dimensions, and measures from a .twbx workbook and re-express them as a production-grade Databricks Metric View YAML — with the Sample Superstore dataset as a complete worked example, accelerated by AI coding...
- 810 Views
- 0 replies
- 1 kudos
- 245 Views
- 0 replies
- 0 kudos
Automating Databricks Lakeflow Connect Pipelines for CDC Databases
Hi all,Tired of paying the data movement tax or wrestling with complex manual pipeline configs?I just published a new Medium article and open-sourced a framework that fully automates Databricks Lakeflow Connect pipelines for CDC-enabled databases usi...
- 245 Views
- 0 replies
- 0 kudos
- 399 Views
- 0 replies
- 1 kudos
Operating PostgreSQL CDC on AWS RDS with Lakeflow Connect
Over the years, I have helped organizations design and deliver large-scale data platforms, and one recurring lesson has remained constant: CDC failures are rarely caused by technology alone. They are usually the result of unclear ownership, missing o...
- 399 Views
- 0 replies
- 1 kudos
- 239 Views
- 0 replies
- 1 kudos
DAIS Community Virtual Challenge 2026: Sysl — Scanning Japanese Receipts
The ProblemLiving in Japan means getting handed receipts everywhere — convenience stores, pharmacies, restaurants. Most end up in a pocket or trash, never tracked, and the coupons go unused.The SolutionSysl is a PWA that scans any Japanese receipt au...
- 239 Views
- 0 replies
- 1 kudos
- 605 Views
- 1 replies
- 2 kudos
DAIS Community Virtual Challenge 2026: LEGO Value Engine - using Data and AI to Find the Best LEGO
Hey everyone!For the DAIS 2026 Community Virtual Challenge, I built a LEGO Value Engine using Databricks Free Edition.This is a passion project that combined my interests of both LEGOs and Data Engineering.When a new LEGO set releases, it can be hard...
- 605 Views
- 1 replies
- 2 kudos
- 2 kudos
GitHub Repository: Lego Value Engine Architecture Diagram (better quality version can be found here)
- 2 kudos
- 486 Views
- 0 replies
- 3 kudos
# Retail demand forecasting on Databricks Free Edition
Hi Databricks Community,I built a retail sales forecasting system on Databricks Free Edition using the Rossmann Store Sales dataset — about 1,115 stores with daily sales over two and a half years. The goal was a 48-day forecast, the same horizon as t...
- 486 Views
- 0 replies
- 3 kudos
- 372 Views
- 0 replies
- 2 kudos
Building a Scalable Data Pipeline with Databricks Free edition | Spark Declarative Pipelines
I recently built an end-to-end data pipeline architecture in the transportation domain, focusing on city and trip data. The pipeline follows the Bronze–Silver–Gold layered approach, where raw data is ingested into the Bronze layer, cleaned and standa...
- 372 Views
- 0 replies
- 2 kudos
- 200 Views
- 0 replies
- 0 kudos
Solving the "Untitled" Lineage Mystery in Unity Catalog
Hey everyone,Have you ever opened Databricks Catalog Explorer to audit a table, only to find the downstream job listed as "Untitled"? Databricks Unity Catalog is incredibly powerful for automated lineage, but it quietly breaks the moment you orchestr...
- 200 Views
- 0 replies
- 0 kudos
- 426 Views
- 0 replies
- 1 kudos
Self Learning Testing Framework using Experiment Tab in Agent Bricks
One of the most valuable capabilities in Agent Bricks is the Experiments tab, where Databricks automatically evaluates different agent configurations and optimization runs using LLM-as-a-Judge techniques. Instead of relying only on traditional metric...
- 426 Views
- 0 replies
- 1 kudos
- 374 Views
- 0 replies
- 0 kudos
Why Your Delta MERGE is 5x Slower Than an Overwrite (And How to Fix It)
Hey everyone,We’ve all been there: a Delta Lake MERGE job that should take 20 minutes drags on for 90 minutes, while a full overwrite of the same table finishes in under 20. When an overwrite outpaces a selective merge, it's a massive red flag that y...
- 374 Views
- 0 replies
- 0 kudos
- 619 Views
- 2 replies
- 4 kudos
How I Passed the Databricks GenAI Engineer Associate — A No-Fluff Study Guide
Hello Everyone, As a Data & Analytics Engineer with experience spanning ETL, data engineering, solution design, and data platform engineering, I currently work Azure Data Ecosystem involving Azure Databricks, Terraform, and CI/CD pipelines — building...
- 619 Views
- 2 replies
- 4 kudos
- 251 Views
- 0 replies
- 0 kudos
Converting stored procedures to PySpark
Hi Everyone,I just publsihed an detailed article in medium for migrating the stored procedures to pyspark #https://medium.com/p/909c5c700ffd?postPublishedType=initial
- 251 Views
- 0 replies
- 0 kudos
- 229 Views
- 0 replies
- 1 kudos
Why We Used Two Bronze Tables Instead of One — And Why It Mattered
Part 1 of a 5-part series on building an enterprise data platform on Databricks.When migrating a large retail conglomerate's SAP HANA platform to Databricks, we needed both historicalcompleteness and near-real-time freshness from day one.That require...
- 229 Views
- 0 replies
- 1 kudos
| User | Count |
|---|---|
| 85 | |
| 75 | |
| 69 | |
| 63 | |
| 44 |
