- 561 Views
- 6 replies
- 11 kudos
Governance RiskOps Agent for Unity Catalog
Body:Every day, data platforms generate thousands of audit events. But here's the problem: security teams are drowning in noise.Critical risks hide in plain sight. Manual investigations take hours. Compliance gaps surface too late. And there's no int...
- 561 Views
- 6 replies
- 11 kudos
- 11 kudos
Hi @WiliamRosa ,Good job! Really interesting use case. Thanks for sharing, and good luck in the contest!
- 11 kudos
- 304 Views
- 1 replies
- 2 kudos
Building an Incremental Customer Data Migration Workflow in Databricks
Building an Incremental Customer Data Migration Workflow in DatabricksBy Naveen AyallaIntroductionIn many enterprise environments, customer data is spread across legacy systems that were originally designed for operational processing rather than mode...
- 304 Views
- 1 replies
- 2 kudos
- 2 kudos
Great write-up, Naveen. Very practical and clear.I really like how you focused not just on migration, but on building a reliable incremental workflow with proper duplicate handling and governance. That’s where real value comes from.Also, connecting D...
- 2 kudos
- 266 Views
- 0 replies
- 1 kudos
From Tableau to Databricks: Migrating KPI Dashboards with Metric Views
How to extract Tableau calculated fields, dimensions, and measures from a .twbx workbook and re-express them as a production-grade Databricks Metric View YAML — with the Sample Superstore dataset as a complete worked example, accelerated by AI coding...
- 266 Views
- 0 replies
- 1 kudos
- 152 Views
- 0 replies
- 0 kudos
Automating Databricks Lakeflow Connect Pipelines for CDC Databases
Hi all,Tired of paying the data movement tax or wrestling with complex manual pipeline configs?I just published a new Medium article and open-sourced a framework that fully automates Databricks Lakeflow Connect pipelines for CDC-enabled databases usi...
- 152 Views
- 0 replies
- 0 kudos
- 157 Views
- 0 replies
- 1 kudos
Operating PostgreSQL CDC on AWS RDS with Lakeflow Connect
Over the years, I have helped organizations design and deliver large-scale data platforms, and one recurring lesson has remained constant: CDC failures are rarely caused by technology alone. They are usually the result of unclear ownership, missing o...
- 157 Views
- 0 replies
- 1 kudos
- 160 Views
- 0 replies
- 1 kudos
DAIS Community Virtual Challenge 2026: Sysl — Scanning Japanese Receipts
The ProblemLiving in Japan means getting handed receipts everywhere — convenience stores, pharmacies, restaurants. Most end up in a pocket or trash, never tracked, and the coupons go unused.The SolutionSysl is a PWA that scans any Japanese receipt au...
- 160 Views
- 0 replies
- 1 kudos
- 363 Views
- 1 replies
- 2 kudos
DAIS Community Virtual Challenge 2026: LEGO Value Engine - using Data and AI to Find the Best LEGO
Hey everyone!For the DAIS 2026 Community Virtual Challenge, I built a LEGO Value Engine using Databricks Free Edition.This is a passion project that combined my interests of both LEGOs and Data Engineering.When a new LEGO set releases, it can be hard...
- 363 Views
- 1 replies
- 2 kudos
- 2 kudos
GitHub Repository: Lego Value Engine Architecture Diagram (better quality version can be found here)
- 2 kudos
- 147 Views
- 0 replies
- 1 kudos
# Retail demand forecasting on Databricks Free Edition
Hi Databricks Community,I built a retail sales forecasting system on Databricks Free Edition using the Rossmann Store Sales dataset — about 1,115 stores with daily sales over two and a half years. The goal was a 48-day forecast, the same horizon as t...
- 147 Views
- 0 replies
- 1 kudos
- 215 Views
- 0 replies
- 2 kudos
Building a Scalable Data Pipeline with Databricks Free edition | Spark Declarative Pipelines
I recently built an end-to-end data pipeline architecture in the transportation domain, focusing on city and trip data. The pipeline follows the Bronze–Silver–Gold layered approach, where raw data is ingested into the Bronze layer, cleaned and standa...
- 215 Views
- 0 replies
- 2 kudos
- 141 Views
- 0 replies
- 0 kudos
Solving the "Untitled" Lineage Mystery in Unity Catalog
Hey everyone,Have you ever opened Databricks Catalog Explorer to audit a table, only to find the downstream job listed as "Untitled"? Databricks Unity Catalog is incredibly powerful for automated lineage, but it quietly breaks the moment you orchestr...
- 141 Views
- 0 replies
- 0 kudos
- 225 Views
- 0 replies
- 0 kudos
Why Your Delta MERGE is 5x Slower Than an Overwrite (And How to Fix It)
Hey everyone,We’ve all been there: a Delta Lake MERGE job that should take 20 minutes drags on for 90 minutes, while a full overwrite of the same table finishes in under 20. When an overwrite outpaces a selective merge, it's a massive red flag that y...
- 225 Views
- 0 replies
- 0 kudos
- 343 Views
- 2 replies
- 4 kudos
How I Passed the Databricks GenAI Engineer Associate — A No-Fluff Study Guide
Hello Everyone, As a Data & Analytics Engineer with experience spanning ETL, data engineering, solution design, and data platform engineering, I currently work Azure Data Ecosystem involving Azure Databricks, Terraform, and CI/CD pipelines — building...
- 343 Views
- 2 replies
- 4 kudos
- 164 Views
- 0 replies
- 0 kudos
Converting stored procedures to PySpark
Hi Everyone,I just publsihed an detailed article in medium for migrating the stored procedures to pyspark #https://medium.com/p/909c5c700ffd?postPublishedType=initial
- 164 Views
- 0 replies
- 0 kudos
- 175 Views
- 0 replies
- 1 kudos
Why We Used Two Bronze Tables Instead of One — And Why It Mattered
Part 1 of a 5-part series on building an enterprise data platform on Databricks.When migrating a large retail conglomerate's SAP HANA platform to Databricks, we needed both historicalcompleteness and near-real-time freshness from day one.That require...
- 175 Views
- 0 replies
- 1 kudos
- 300 Views
- 0 replies
- 0 kudos
Stop giving your Databricks AI Agents 50 tools to manage
If you are building enterprise Generative AI, you know the pain of the "Monolithic Agent Bottleneck": passing dozens of tools to a single AI supervisor leads to hallucinated routing, massive context bloat, and security nightmares.With the recent May ...
- 300 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 85 | |
| 74 | |
| 57 | |
| 44 | |
| 44 |
