- 722 Views
- 1 replies
- 2 kudos
DAIS Community Virtual Challenge 2026: LEGO Value Engine - using Data and AI to Find the Best LEGO
Hey everyone!For the DAIS 2026 Community Virtual Challenge, I built a LEGO Value Engine using Databricks Free Edition.This is a passion project that combined my interests of both LEGOs and Data Engineering.When a new LEGO set releases, it can be hard...
- 722 Views
- 1 replies
- 2 kudos
- 2 kudos
GitHub Repository: Lego Value Engine Architecture Diagram (better quality version can be found here)
- 2 kudos
- 665 Views
- 0 replies
- 3 kudos
# Retail demand forecasting on Databricks Free Edition
Hi Databricks Community,I built a retail sales forecasting system on Databricks Free Edition using the Rossmann Store Sales dataset — about 1,115 stores with daily sales over two and a half years. The goal was a 48-day forecast, the same horizon as t...
- 665 Views
- 0 replies
- 3 kudos
- 478 Views
- 0 replies
- 2 kudos
Building a Scalable Data Pipeline with Databricks Free edition | Spark Declarative Pipelines
I recently built an end-to-end data pipeline architecture in the transportation domain, focusing on city and trip data. The pipeline follows the Bronze–Silver–Gold layered approach, where raw data is ingested into the Bronze layer, cleaned and standa...
- 478 Views
- 0 replies
- 2 kudos
- 234 Views
- 0 replies
- 0 kudos
Solving the "Untitled" Lineage Mystery in Unity Catalog
Hey everyone,Have you ever opened Databricks Catalog Explorer to audit a table, only to find the downstream job listed as "Untitled"? Databricks Unity Catalog is incredibly powerful for automated lineage, but it quietly breaks the moment you orchestr...
- 234 Views
- 0 replies
- 0 kudos
- 482 Views
- 0 replies
- 1 kudos
Self Learning Testing Framework using Experiment Tab in Agent Bricks
One of the most valuable capabilities in Agent Bricks is the Experiments tab, where Databricks automatically evaluates different agent configurations and optimization runs using LLM-as-a-Judge techniques. Instead of relying only on traditional metric...
- 482 Views
- 0 replies
- 1 kudos
- 418 Views
- 0 replies
- 0 kudos
Why Your Delta MERGE is 5x Slower Than an Overwrite (And How to Fix It)
Hey everyone,We’ve all been there: a Delta Lake MERGE job that should take 20 minutes drags on for 90 minutes, while a full overwrite of the same table finishes in under 20. When an overwrite outpaces a selective merge, it's a massive red flag that y...
- 418 Views
- 0 replies
- 0 kudos
- 1050 Views
- 2 replies
- 4 kudos
How I Passed the Databricks GenAI Engineer Associate — A No-Fluff Study Guide
Hello Everyone, As a Data & Analytics Engineer with experience spanning ETL, data engineering, solution design, and data platform engineering, I currently work Azure Data Ecosystem involving Azure Databricks, Terraform, and CI/CD pipelines — building...
- 1050 Views
- 2 replies
- 4 kudos
- 294 Views
- 0 replies
- 0 kudos
Converting stored procedures to PySpark
Hi Everyone,I just publsihed an detailed article in medium for migrating the stored procedures to pyspark #https://medium.com/p/909c5c700ffd?postPublishedType=initial
- 294 Views
- 0 replies
- 0 kudos
- 289 Views
- 0 replies
- 1 kudos
Why We Used Two Bronze Tables Instead of One — And Why It Mattered
Part 1 of a 5-part series on building an enterprise data platform on Databricks.When migrating a large retail conglomerate's SAP HANA platform to Databricks, we needed both historicalcompleteness and near-real-time freshness from day one.That require...
- 289 Views
- 0 replies
- 1 kudos
- 639 Views
- 0 replies
- 0 kudos
Stop giving your Databricks AI Agents 50 tools to manage
If you are building enterprise Generative AI, you know the pain of the "Monolithic Agent Bottleneck": passing dozens of tools to a single AI supervisor leads to hallucinated routing, massive context bloat, and security nightmares.With the recent May ...
- 639 Views
- 0 replies
- 0 kudos
- 384 Views
- 0 replies
- 1 kudos
DAIS Virtual Challenge 2026: Weather-Aware Event Recommendation & Safety Advisory using Databricks
I'm exploring an idea that combines event discovery, weather intelligence, and safety awareness into a single AI-powered experience built on Databricks. The ProblemWhen people travel or plan activities, they often face three common challenges:Activit...
- 384 Views
- 0 replies
- 1 kudos
- 480 Views
- 0 replies
- 0 kudos
Databricks SQL Just Dropped Some Massive Engine Upgrades for Data Engineers
While AI and LLMs take the headlines, hardcore data engineers know that SQL remains the operational backbone of enterprise pipelines. Databricks just rolled out several powerful programmatic and geospatial updates that solve real-world, complex data ...
- 480 Views
- 0 replies
- 0 kudos
- 381 Views
- 0 replies
- 1 kudos
Building an AI Powered Autonomous Data Reliability Platform using Databricks & Gemini LLM
What if a data pipeline could explain why, it failed instead of just saying it failed? While learning Databricks and exploring Data Engineering, I built an AI Powered Autonomous Data Reliability Platform on Databricks Free Edition using: PySpark Delt...
- 381 Views
- 0 replies
- 1 kudos
- 820 Views
- 0 replies
- 3 kudos
Deal2Delivery: How I Built an End-to-End AI Sales Intelligence Platform on Databricks
Hi Everyone! This is my official submission for DAIS 2026 Community Virtual Contest!Deal2Delivery: How I Built an End-to-End AI Sales Intelligence Platform on DatabricksEvery sales team has the same nightmare: a deal closes, and then nobody knows if ...
- 820 Views
- 0 replies
- 3 kudos
- 453 Views
- 0 replies
- 1 kudos
Your Delta Lake Table Is Secretly Ballooning — Here's the 2-Command Fix
Let Me Guess What Happened to YouYou built a solid data pipeline. It runs every day, ingests a few gigabytes, everything looks fine. Then one morning you open your cloud storage bill — AWS S3, Azure ADLS, or Google Cloud Storage — and something feels...
- 453 Views
- 0 replies
- 1 kudos
