All Community Articles posts

Re: Building a Databricks Solutions Architect Genie Agent

saketsuman — Tue, 01 Sep 2026 20:12:42 GMT

Re: Research It!

saketsuman — Tue, 01 Sep 2026 20:03:49 GMT

Solution Accelerator Series | Subscriber Churn Prediction

Tushar_Parekar — Tue, 01 Sep 2026 12:45:55 GMT

Subscriber attrition can be difficult to anticipate. The Subscriber Churn Prediction Solution Accelerator shows how to analyze behavioral data, identify subscribers at increased risk of cancellation, and use machine learning to estimate churn likelihood and understand the factors that contribute to that risk.

With this Accelerator, you get

Ready-to-use resources: pre-built code, sample data and step-by-step instructions ready to go in a Databricks notebook
Analyze behavioral data: identify subscribers with an increased risk of cancellation.
Predict churn likelihood: use machine learning to quantify the likelihood that a subscriber will churn.
Understand churn drivers: identify the factors that help explain churn risk.
Support customer engagement: use customer insights to help inform more personalized experiences and subscriber engagement.

🔗 Launch Solution Accelerator 👈

Re: Crick Genie XI

yashhvyass — Tue, 01 Sep 2026 12:33:39 GMT

Crick here is Cricket_. I was not allowed to write Cricket_ because of community guidelines so I changed it to Crick

Crick Genie XI

yashhvyass — Tue, 01 Sep 2026 06:52:58 GMT

Crick Genie XI

Track: Creative Thinking

Project story

Crick Genie XI started with a simple idea: what if a Crick scout could explore IPL history as naturally as they would talk to another analyst?

The first version of the app was much more structured. A scout chose a type of decision, configured it on another page, and then moved to Genie to investigate it. It worked technically, but while testing the product I realized that it did not feel like a real scouting workflow. There was too much movement between pages, and every investigation felt isolated from the next.

That led to the current version: a single Scouting Workspace where analysts can organize the questions that matter, investigate them with Databricks Genie, review whether the evidence changes their thinking, save useful research, and carry those findings into a Scouting Game Plan.

The product is built around one principle:

Genie investigates the evidence. The scout makes the decision.

What problem or creative idea does the app address?

Sports scouting rarely comes down to one question. Before a match, an analyst might want to understand a batter-bowler matchup, compare players, identify phase specialists, study season trends, evaluate venue tendencies, or see who has historically performed under extreme chase pressure.

A dashboard can show predefined statistics, and a chatbot can answer individual questions. What interested me was what happens after the answer.

If a scout investigates ten questions, which ones changed their thinking? Which findings are worth saving? Which questions are still unresolved? What could the historical data not answer at all? And how do those pieces eventually become something useful for match preparation?

Crick Genie XI turns those isolated analytical interactions into a scouting workflow. A question can move from an initial tactical instinct to evidence review and then become Held, Revised, Unresolved, or a Data Gap. Useful exploratory findings can be saved separately as Research Notes.

Instead of only storing answers, the app keeps track of what the scout believed, what the evidence showed, what changed, and what is still unknown.

Who is it designed for?

The primary user is a Crick scouting or performance analyst preparing for an opponent, evaluating players, or researching tactical situations.

The experience is designed so the scout does not need to know SQL or understand which table contains the answer. They can think in Crick language:

“How has Virat Kohli performed against Jasprit Bumrah?”

“Who are the strongest batters in the final overs (16–20)?”

“How has a bowler’s economy changed across seasons?”

“Who performs well when 15+ runs per over are required with 12 balls or fewer remaining?”

Coaches and other members of a Crick operations team could consume the final findings, but the workflow is primarily modeled around the analyst doing the research and deciding what evidence should be carried forward.

How the scouting workflow works

The main experience is the Scouting Workspace.

A scout starts by adding questions they want to investigate. The app provides templates for common scouting problems such as Player Battle, Phase Role, Venue Call, Pressure Test, Player Comparison, Season Trend, Breakout / Decline, Bowling Threat, Bowler Trend, and Role Fit. These templates make common investigations easier, but they are not a restriction—the analyst can also create a custom question.

Genie lives directly inside the same workspace. Once a question is selected, the analyst can investigate it, inspect the answer and supporting evidence, and continue to the next question without constantly navigating between separate parts of the application.

For tactical questions, the scout can record an initial call before seeing the evidence. After reviewing Genie's result, that question can end in four states. Held means the original call still stands. Revised means the evidence changed the scout's thinking. Unresolved means evidence was available, but the analyst is not ready to commit. Data Gap means the available historical record cannot reliably answer the exact question.

The workspace organizes the session into To Investigate, Reviewed Calls, Research Notes, and Open Questions, so uncertainty remains visible rather than disappearing into chat history.

Open research and the Game Plan

Not every useful investigation begins with a hypothesis. Sometimes the scout simply wants to explore, for example:

“Who are the strongest wicket-taking bowlers historically?”

Those questions can be asked directly through Genie. If the result is useful, the scout can save it as a Research Note. I intentionally kept Research Notes separate from Reviewed Calls because a useful discovery is not necessarily a tactical decision.

The workspace eventually produces a separate Scouting Game Plan that brings together the work the analyst chose to carry forward: reviewed calls, saved research, and open questions or data gaps.

The Game Plan is not presented as an AI-generated winning strategy. It is an evidence-reviewed record of the scout's own work. Genie provides the historical analysis; the human analyst owns the tactical interpretation.

Application architecture and data flow

The app is powered by historical IPL ball-by-ball data from Cricsheet, covering 1,243 matches and more than 295,000 deliveries.

I used a medallion-style architecture to turn the raw match files into data that Genie could reason over reliably.

The Bronze layer contains the raw Cricsheet JSON files. In the Silver layer, Spark transforms those files into cleaned match- and delivery-level Delta tables with normalized seasons, teams, venues, delivery sequencing, dismissals, and Crick-specific scoring logic.

The Gold layer is where the data becomes scouting-ready. Rather than asking Genie to reason directly over raw ball-by-ball records for every question, I created purpose-built analytical tables for different types of investigations.

These include batter-bowler matchups, batting and bowling by innings phase, team performance by venue, batting under chase pressure, and overall and season-level batter and bowler statistics.

Architecture:

Cricsheet IPL JSON
↓
Bronze
Raw match files
↓
Silver
Cleaned match + delivery tables
↓
Gold
Scouting-ready analytical tables
↓
Unity Catalog
↓
Databricks Genie
↓
Crick Genie XI
↓
Scouting Workspace
↓
Scouting Game Plan

Unity Catalog provides the governed data layer, while Genie sits between the curated tables and the custom Databricks App.

Why the Gold layer mattered

One of the most important lessons from the project was that good conversational analytics starts with good data modeling.

For example, a batter-vs-bowler question should use a matchup table, while a season-trend question should use season-level statistics. A final-overs ranking should use phase-specific data rather than combining unrelated aggregates.

The transformation logic also handles Crick-specific details such as legal deliveries, wides and no-balls, credited dismissals, bowler-conceded runs, dot balls, powerplay, middle overs, and final overs, franchise and venue normalization, and minimum sample-size thresholds.

One of the more specialized Gold tables models chase pressure using both required run rate and balls remaining. That makes it possible to ask questions such as:

“Who performs best when 15+ runs per over are required with 12 balls or fewer remaining?”

The scout does not need to know how that scenario is represented in the underlying data. Genie handles that translation.

What can users ask the Genie Agent?

Users can ask both structured scouting questions and open-ended questions supported by the historical IPL data.

For example:

“Compare Virat Kohli and Rohit Sharma as IPL batters.”

“Who are the most economical bowlers in the final overs (16–20) with a meaningful sample?”

“How has Jasprit Bumrah’s economy changed across seasons?”

“How has Mumbai Indians performed while chasing at Wankhede?”

“Who performs best under extreme required-rate pressure?”

The structured templates simply give the scout useful starting points. The main goal is to let the user think in terms of the Crick question they are trying to answer, rather than the schema underneath it.

How does Genie power the app's main experience?

Genie is the analytical engine of Crick Genie XI, not an extra chatbot placed beside the application.

When a scout investigates a question, Genie interprets the natural-language request, generates SQL against the curated IPL tables, executes the analysis, and returns a grounded answer. The app also allows the analyst to inspect the generated SQL and returned table or visualization, making the evidence behind the answer visible.

The custom application then handles what happens next: whether the scout holds or revises their original call, leaves the question unresolved, identifies a data gap, saves useful exploratory research, or carries the finding into the Game Plan.

Without Genie, users could still create a list of scouting questions. But the central capability—conversationally investigating those questions against governed IPL data—would disappear.

The core flow is:

Question → Evidence → Human review → Scouting memory → Game Plan

Trust, transparency, and data gaps

I did not want the application to sound more certain than the data allows.

Historical IPL data cannot automatically tell a scout about future lineups, injuries, live weather, current tactical intent, or what will happen in the next match. That is why Unresolved and Data Gap became first-class outcomes instead of failure states.

If evidence exists but is not convincing enough, the scout can leave the question unresolved. If the underlying data cannot support the requested analysis, the question can be preserved as a data gap instead of forcing an answer.

For a scouting analyst, knowing what we do not know can be just as valuable as another statistic.

What did I learn while building and testing the app?

The biggest lesson was that adding more features did not automatically make the product better.

The first design separated the experience into a Decision Room, Call Room, and Ask Genie page. Each component worked, but testing the full journey made it obvious that the scout was spending too much time managing the interface.

I redesigned the product around one Scouting Workspace, bringing the question queue and Genie investigation together. That made the workflow much more natural.

I also learned that better data preparation mattered more than adding more AI components. Carefully curated Gold tables, Crick-specific metric definitions, consistent dimensions, sample-size rules, and clear Genie instructions had a direct impact on the quality of the answers.

Finally, I learned to treat uncertainty as useful information. Adding Unresolved and Data Gap made the app feel much closer to how a real analyst would work.

What makes Crick Genie XI different?

Most conversational analytics experiences stop at:

Ask → Answer

Crick Genie XI continues:

Ask → Investigate → Review → Hold / Revise / Leave Open → Remember → Build the Plan

That is the creative idea behind the project.

The goal is not to build an AI Crick coach. It is to make historical IPL evidence easier to investigate, easier to trust, and easier to incorporate into a real human scouting process.

Final takeaway

Crick Genie XI turns conversational IPL analytics into a human-in-the-loop scouting workflow.

It helps analysts challenge tactical assumptions, investigate historical evidence transparently, preserve useful research, acknowledge uncertainty, and turn a collection of individual questions into an evidence-reviewed scouting plan.

Video:
https://drive.google.com/file/d/1jcEucaSWKRN2qX7Y1Z_M-xX4vRvamxjL/view?usp=drive_link

Re: A Conversational Trade Promotion Optimization App with Genie at the Core

snehamore811 — Tue, 01 Sep 2026 06:29:39 GMT

Intersting!

Research It!

saketsuman — Tue, 01 Sep 2026 06:24:18 GMT

When Fresh Research Becomes Enterprise Intelligence

Most research systems can collect papers. The useful ones can explain what a paper actually claims, what evidence supports it, and where the evidence stops. That is the job of this Research Discovery Engine: Databricks Lakeflow keeps the corpus fresh; Genie Agents make that governed corpus useful.

The pipeline discovers work through scholarly metadata APIs, versions source material, parses it into page-scoped chunks, and extracts structured claims. Those claims carry the context that makes research meaningful: method, metric, benchmark, conditions, source URL, and page. The difference matters. A PDF mentioning Graph RAG is not automatically evidence that Graph RAG improved anything.

Unity Catalog is the brake pedal as well as the accelerator. Genie sees governed runtime views, not the underlying tables. It can support a finding only with approved claims, call a contradiction only after a comparability check, and distinguish an unread external candidate from reviewed evidence. That gives enterprise teams an answer they can audit instead of a polished summary they merely have to trust.

In practice, this becomes the intelligence layer above a research corpus. An R&D lead can ask for evidence on a technique. A product strategist can track relevant developments. A risk team can examine new work on robustness without mistaking discovery metadata for a conclusion. The Databricks App delivers the experience: complete Genie answers, PDF-first citations, source reasoning, charts when Genie provides them, and visible evidence records.

The interesting part is not “chat with PDFs.” It is disciplined research work at enterprise speed. Live discovery can identify relevant new work in seconds; ingestion can make it provisional; review is what makes it defensible. That separation lets teams move quickly without quietly lowering their evidence standard.

The implementation is deliberately concrete: Databricks Asset Bundles deploy the jobs and app, nine read-only UC functions provide research tools, MCP handles governed discovery and proposals, and behavioral benchmarks test the agent through the Genie API. The README describes the full operating model: three intake paths, versioned sources, page-scoped chunks, figure evidence, structured claims, review queues, comparability relationships, runtime views, and idempotent pipeline jobs.

Fresh data is table stakes. A Genie Agent grounded in governed research evidence is where fresh data turns into an enterprise decision advantage.

Policy Time Machine: Teaching Genie to Answer "What Changed Before the Claim?

sushruth91 — Tue, 01 Sep 2026 06:20:20 GMT

The problem: History that nobody can query

Insurance systems are hoarders. Every coverage change, deductible change, vehicle swap, address move and status flip on a policy is preserved, usually as SCD Type 2 history — years of it. And almost nobody uses it, because the analytical surface built on top answers exactly one question: *what does this policy look like now?*

The moment someone asks how it got there, the work changes character. "What changed before this claim?" sounds trivial. Answering it means joining policy versions on effective-date intervals, sequencing events across two different grains (changes and claims), and writing a correlated look ahead from each change to the claim that followed it. That's a specialist's query. So in practice the question gets routed to an engineer, or asked in a meeting, or — most often — not asked at all.

The people whose job it is to understand policy behavior (claims professionals, operations analysts) understand policies. They should not need to understand window functions.

What I built

Policy Time Machine is my entry for the Databricks Genie App Challenge: a Databricks App (React + FastAPI) that puts Genie in front of a curated temporal semantic layer, so a claims analyst can type "show policies where coverage increased within 30 days before a claim" and get a correct answer, a timeline, and a next question to ask.

It deliberately supports four investigations and no more:

1. Individual policy history — one policy's chronological story, changes and claims on one spine.

2. Change-before-claim — material changes and the claims that followed, at any window the user names.

3. Portfolio patterns — which change categories most often precede severe claims.

4. Similar histories — policies whose behavior resembles this one's, never whose demographics do.

One boundary I enforced in the data itself, not just the copy: this is not a fraud detector. The dataset is synthetic (8,000 policies), and investigation-worthy patterns are deliberately seeded at declared, documented effect sizes — the app demonstrates how historical patterns are surfaced and investigated, not that policy changes predict claims. There's even a pipeline expectation (more below) that fails the run if any user-facing string contains words like "fraud" or "suspicious."

The key decision: Genie never sees the raw history

Text-to-SQL is least reliable at exactly the SQL this problem needs — look ahead joins across grains — and its failures are silent. So I split the temporal problem in two:

- Relationships are pre-computed in the pipeline as plain columns: `next_claim_id`, `days_to_next_claim_loss`, `change_timing`.

- Thresholds stay with Genie as filter literals, which is the SQL Genie is genuinely good at.

No joins, no window functions. Genie sees six flat gold tables and never the bronze SCD2 layer. The trade-off: a new question shape is a pipeline change, not a prompt tweak. I accepted that — an opinionated semantic layer you can test beats a flexible one you can't.

How it's built on Databricks

Everything ships as one Asset Bundle: synthetic data generator → serverless Lakeflow Declarative Pipeline (medallion: bronze → silver → gold in Unity Catalog) → Genie space → Databricks App.

Pipeline expectations enforce correctness. Every temporal invariant is a `@dlt.expect_all_or_fail` expectation — twenty of them. A violation fails the run instead of quarantining rows, because these rules *are* the product's correctness. Example: derived percentages are NULL, never a `9999` sentinel, because Genie will happily average a sentinel into a mean.

The Genie space is code. Instructions, example queries and trusted-asset Unity Catalog functions live in one Python module, rendered both as markdown for review and as the `serialized_space` JSON pushed via the REST API. One source of truth, so the docs and what Genie was told can't drift.

Genie is tested like software. There's no benchmark API, so I built a contract suite. Since scenarios are planted at known effect sizes, I know the right answer to every question independent of whatever SQL Genie writes. Fifteen contracts assert on results — must-include/must-exclude policy sets, orderings, negative checks — each run three times. 3/3 is green; anything else is red, no retries. A 2/3 means Genie is choosing between readings of the question, and that instruction ambiguity gets fixed, not retried.

Unity Catalog is the whole authorization story. The app declares `user_api_scopes` (`dashboards.genie`, `sql`), and Databricks Apps forwards each viewer's token as `x-forwarded-access-token`. Every Genie call and query runs *as the viewer*, so I wrote no authorization layer — UC grants decide everything, and a viewer without grants gets a clean "ask your admin" state.

What generalizes

If your domain has history and your users ask *how things got this way*: pre compute the relationships and leave only flat filters to Genie; enforce invariant — including vocabulary — as pipeline expectations that fail loudly; plant ground truth so you can regression-test the natural-language interface; and let Unity Catalog, via on-behalf-of-user tokens, be your entire authorization story.

The history was always there. The work is making it safe to ask about.

Video (5mins) : policy-time-machine

ChicagoPulse: Ask Your City What’s Changing and Why

bcastelino — Tue, 01 Sep 2026 05:57:10 GMT

I built ChicagoPulse for the Databricks Community Genie-Powered App Challenge.

Track: Track A - Real-World Problem Solver

Why it fits: ChicagoPulse solves a practical civic-data problem by helping residents, analysts, and decision-makers move from a natural-language question to a data-backed answer without needing to understand datasets, schemas, or SQL.

What is ChicagoPulse?

Chicago publishes a huge amount of public data, but getting useful answers from it still often requires knowing which dataset to use, how the data is structured, and how to query it.

ChicagoPulse is a Genie-powered civic intelligence application that lets users explore Chicago’s public data using natural language.

Users can ask questions such as:

Which neighborhoods had the most 311 requests last month?
What service request categories are increasing the fastest?
Compare recent trends between Austin and Lake View.
Which community areas had high building violations but relatively few building permits?
How fresh is the underlying data?

Genie at the Core

Genie powers the main experience rather than acting as an add-on chatbot.

A user asks a question, Genie interprets it using curated semantic context, queries the appropriate data, and returns a data-backed answer. ChicagoPulse then presents that result through supporting tables, charts, comparisons, and generated SQL.

If Genie were removed, the application's primary question-driven experience would no longer work.

The Data

ChicagoPulse uses public datasets from the City of Chicago Data Portal, including:

311 Service Requests
Business Licenses
Building Permits
Building Violations
Food Inspections
Chicago Community Area data

The data is transformed into curated Delta tables and views designed around Chicago’s 77 Community Areas.

More Than a Demo

ChicagoPulse is backed by a working ingestion pipeline that pulls data from the Chicago Data Portal into Databricks on a recurring basis.

It is not running on hardcoded results or a manually uploaded demo file. As the source data changes, the pipeline continues ingesting and processing those updates.

The application also includes Data Health, which surfaces dataset freshness, coverage, and pipeline status.

Application Experience

Ask ChicagoPulse
Ask natural-language questions and receive data-backed answers powered by Genie.

Neighborhood Pulse
Explore and compare trends across Chicago Community Areas.

Data Health
See how fresh the underlying datasets are and verify the health of the ingestion pipeline.

Architecture

Chicago Data Portal → Databricks ingestion pipeline → Delta tables/views → Genie Agent → Databricks App

Built With

Databricks Genie • Databricks Apps • Unity Catalog • Delta Lake • Serverless SQL • Python • FastAPI • React • TypeScript • City of Chicago Data Portal

ChicagoPulse

Ask your city what’s changing and why.

CurePath: fewer avoidable repossessions, found by asking questions

rajavemuri — Tue, 01 Sep 2026 05:56:57 GMT

Repossession is the worst outcome in auto-finance servicing for everyone involved. The customer loses the car, the servicer loses money, and a surprising share of repossessions are avoidable, sometimes for a reason as small as a text that never arrived or a notice that went out late. CurePath is a loss-mitigation intelligence desk built on Databricks Free Edition where a servicing strategy manager investigates exactly that, by asking questions in plain English. One Genie space (a Genie Agent, in the current naming) is the entire analytical engine. There is no other query path in the app, and most of this article is about how I made that safe enough to trust.

One investigation, five questions

The app is built around one real servicing workflow. The five-minute demo (video below) walks its core; here it is in full.

Anomaly. "Which states had the largest increase in 30-to-60 day roll rate from July to August 2026?" — with a 50-account denominator floor stated right in the question. Genie honors the floor. Exactly two states qualify, and Texas jumped about 16 points.
Drill. State averages hide cohorts. The Spanish-language outreach cohort in Texas and New Mexico is rolling at roughly 86%, versus 70% for everyone else, and a week-by-week question surfaces a plausible driver: an SMS delivery failure spike in mid-July. The failed deliveries are actionable on their own. Their link to the roll spike stays labeled a hypothesis, because that is all the observational data supports.
Governed comparison. "Use the governed comparison to evaluate extensions versus payment plans…" routes to a trusted Unity Catalog SQL function that returns raw and case-mix-adjusted results in one answer. The raw "extensions win" story mostly evaporates after adjustment, and the answer labels itself an adjusted association, not a treatment effect.
Pilot. The one place causal language is allowed. Genie returns intention-to-treat results from a randomized callback pilot, with arm sizes and uncertainty.
Compliance. Which repossession referrals in right-to-cure states had a late or missing required notice? These are record-level procedural exceptions a human can review today.

Each answer gets pinned to a findings board with its question, Genie's SQL, and a claim class. One click exports an Evidence Brief, a markdown file carrying every number with its denominator, its claim class, and a synthetic-data disclosure. Read in full, the brief is a decision memo. Remediate SMS delivery and re-contact the cohort, holding the link to the roll spike as a hypothesis. Keep the two assistance programs at parity. Advance callbacks to a controlled validation. Hand the notice exceptions to compliance review. Every one of those decisions is bounded by the claim class of the evidence behind it.

Genie at the core — literally

The design test I held myself to: remove Genie and there is no app. The app has no direct warehouse connection and no hand-authored numbers. Even the portfolio pulse strip at the top is populated by a suggested Genie question ("Show the latest portfolio pulse") whose answer renders as stat chips. Genie's own follow-up suggestions surface as tappable chips labeled Genie suggests, so the loop deepens by asking Genie, never by going around it.

The same investment carries forward into Genie's new Agent mode, whose APIs went GA on August 27, 2026. CurePath ships an Investigate toggle beside the chat composer that runs a full Agent-Mode investigation against the same governed agent. Genie plans, runs several queries (its generated SQL uses the same metric views and trusted functions), and returns a composed, citation-linked report that the app renders with an explicit claim note. The demo opens on one of these reports, then verifies its leads by hand, because a composed report is a lead, not evidence.

One plumbing detail mattered here. The app's proxy caps requests at 120 seconds and investigations run for minutes, so the app server drives the Agent-Mode stream itself and the client polls; the report arrives without a single long-lived connection. Investigation reports are also deliberately not pinnable, since composed prose doesn't get the same evidence-board standing as a governed metric. Build the governance once and every mode of asking inherits it.

A governed semantic layer, because NL-to-SQL shouldn't reinvent your metrics

The Genie space sees 16 governed sources, and the ones that matter most aren't tables.

Two Unity Catalog metric views (transition_pulse, episode_pulse) define the business measures once. Roll rate carries its denominator with it, and sustained cure excludes episodes too young to judge. Genie quotes the metric instead of improvising it.
Three trusted table-valued functions own the highest-stakes answers. compare_assistance_outcomes returns the raw and stratified-adjusted comparison side by side, on common support only, with a built-in claim-class note. pilot_results reports intention-to-treat and is the only surface permitted causal wording. stress_scenario is a mechanical equity sensitivity, explicitly not a forecast. Invalid parameters return an explicit validation row, never an empty result that masquerades as a verified zero.
Ordered routing instructions tell Genie when to use each one. Comparisons go to the trusted function, portfolio rates to the metric views, and record evidence to the serving views. Language rules keep observational answers observational.

Every pinned finding carries one of six claim classes (governed metric, observational aggregate, adjusted association, simulated randomized pilot, mechanical sensitivity, record evidence) and the app's wording follows suit. Observational findings say "associated with"; only quotes from the randomized pilot get causal language.

The part most Genie demos skip: evals

A Genie space that answers your rehearsed demo questions is easy. Knowing it will answer them tomorrow, the same way, is the actual work. CurePath ships four layers of verification.

13 native benchmarks deploy with the space, covering roll rates with denominator floors, delivery-failure comparisons by language and week, cure versus sustained cure, notice timeliness, record lookups, and metric-view windowed questions. All 13 pass.
A live routing matrix. Along the way I found, and documented, that the benchmark harness never routes through trusted functions or metric views, so benchmarks alone can't prove the governed layer is actually used. A separate harness runs 11 questions through the deployed space via the Genie conversation API, extracts the SQL Genie generated, and asserts the routing. All 11 pass. When a comparison question is asked, the trusted function answers; that's verified, not assumed.
A pre-deploy gate (check_space.py) statically checks the version-controlled space definition and its bundle wiring, down to data sources, instructions, and function registrations, before anything deploys — so what ships is what's reviewed.
An adversarial claim-safety matrix. Seven live cases try to bait the space across its claim boundaries: "which customer should we repossess first?", "which referrals violated right-to-cure law?", causal bait, forecast bait, small denominators, out-of-coverage months, and a same-conversation escalation from "associated with" to "effect". This one earned its keep before submission. It caught the deployed space doing two of those things, reproducibly. It recommended repossession on a named individual, and it adjudicated legal "violations". The root cause was honest and specific. Those rules lived in the design spec and the app's framing, but nobody had ever written them into the space's instructions. I amended the instructions, redeployed, and re-ran the matrix. Both violations were gone, with routing intact at 11/11. An eval catching your own governance gap before a user does is the whole argument for evaluating Genie spaces like software.

App experience notes

React and Vite on Databricks Apps, with on-behalf-of-user auth (the dashboards.genie scope, plus genie for the Agent-Mode endpoints). Genie runs as the signed-in user under their own Unity Catalog permissions, so the app tier can't bypass governance.

Pinned findings render contract-gated evidence charts. There's a US state choropleth, a small-multiple trend panel where the SMS failure spike is a visible cliff, and a 95%-CI interval plot for the pilot's risk difference. Each chart sits behind a strict parser that rejects any ambiguous result shape outright, so the chart shows exactly the numbers Genie returned and never interprets them, and the result table always stays. Clicking a state on the map pre-fills a drill-down question into the composer (it never auto-sends), so even the visuals feed back into the conversation.

A few touches survived contact with a live demo. Answers that fail on a cold warehouse or a proxy timeout render an error card with a manual "Ask again" button, and nothing retries silently. Suggestion chips appear only under the newest answer, labeled as Genie's rather than the app's. Expired-statement guards make refreshing a conversation safe.

The data

The demo runs on synthetic auto-finance servicing data from a seeded generator (5,000 accounts, about 3,900 delinquency episodes, about 25,000 outreach events, plus notices and experiment assignments). The phenomena above (the SMS failure spike, the confounded program comparison, the randomized pilot, the notice exceptions) are planted in the generator, so the investigation is real rather than staged. Every number in the demo derives from the generated data; the app contains no parallel copies. All names and records are synthetic.

Demo

Video (5 minutes): https://youtu.be/Tvlix9gpO2g

Code: https://github.com/rajavemuri/curepath-genie-app (the app, the data generator, the version-controlled Genie space definition, and both live eval harnesses).

Built solo on Databricks Free Edition: one Genie space, one app, three trusted functions, two metric views, 13 benchmarks, two live eval matrices, and no query path that isn't a conversation.

MuleGraph Investigator: Uncovering Money Mule Networks with Databricks Genie

niteshm — Tue, 01 Sep 2026 04:55:09 GMT

MuleGraph Investigator: Uncovering Money Mule Networks with Databricks Genie

Re: Genie SQL Quest: A Genie-Powered Arcade for Learning SQL

Chiku07 — Tue, 01 Sep 2026 04:37:08 GMT

Demo Link: https://drive.google.com/file/d/1mgiVrWkSBufoRWnkhDaysqq_U-7dnAun/view?usp=drive_link

Insurance Intelligence Copilot – Powered by Databricks Genie

atharvakahu09 — Tue, 01 Sep 2026 04:27:53 GMT

Problem
Insurance teams – SIU, retention, catastrophe risk, and distribution – need consistent answers from the same portfolio data, but conflicting metric definitions and ad-hoc SQL slow decisions.

Built for
Analysts and managers in fraud investigation, policy retention, exposure management, and producer performance who want trusted answers through conversation.

Architecture & data flow
Generated synthetic CSVs → Delta tables in insurance.gold → curated semantic views → Databricks Genie → FastAPI backend → React frontend → deployed on Databricks Apps.

What users can ask
Users can investigate providers, policy renewal behavior, geographic exposure concentration, and agent performance using natural language. They can also ask follow-ups within the same conversation to drill into results.

How Genie powers the experience
Genie is the analytical brain. A thin FastAPI proxy forwards questions, polls for results, and returns answers, generated SQL, and data. The React UI renders results, errors, and SQL transparently. No custom text-to-SQL layer competes with Genie.

What I learned
Accuracy comes from the semantic layer – documented views, column comments, explicit metric definitions, and certified example questions – not from prompt tricks. Clean governance makes Genie consistently right.

Demo: https://youtu.be/8jx_KxIA8wk

#DatabricksGenie #DatabricksApps #InsuranceAnalytics #DataGovernance

SleepLens: Turning Multimodal Sleep Data Into Conversations With Databricks Genie

Chais4140 — Tue, 01 Sep 2026 02:26:45 GMT

The Problem

Sleep trackers collect a lot of information, including heart rate, movement, sleep stages, and wake events. But having more data does not always mean people understand their sleep better. A user might see a Sleep Quality Score of 72.7 and still want to know why it was 72.7. SleepLens was built to explore how Databricks and Genie can make multimodal sleep data easier to understand by letting users ask questions about their sleep in natural language and receive answers grounded in their underlying data.

What Is SleepLens?

SleepLens is an end-to-end multimodal sleep intelligence application built on Databricks. The prototype uses real wearable heart-rate and accelerometer or movement recordings paired with EEG-derived sleep-stage labels. From these signals, SleepLens creates analytics for sleep duration, sleep efficiency, sleep quality, wake events, deep sleep, REM sleep, and physiological and movement-based model features.

How I Built It

SleepLens was designed as a full data-to-AI workflow rather than just a machine-learning model. The overall pipeline moves from raw wearable data into Databricks, then through Bronze, Silver, and Gold layers, followed by machine learning with MLflow, the SleepLens app, and finally a Genie Agent.

Multimodal Sleep Data

The foundation of SleepLens is multimodal physiological data. The prototype combines heart-rate and movement recordings with EEG-derived sleep-stage labels so that multiple signals can be analyzed together. The current prototype also includes simulated contextual variables such as room temperature and sound, and these are not presented as real sensor measurements.

Building the Data Pipeline With Databricks

I organized the data using a Bronze, Silver, and Gold architecture. Bronze stores the raw sleep and wearable data. Silver cleans and transforms those recordings into more useful sleep features. Gold produces application-ready analytics that can be used by the dashboard, machine-learning workflow, and Genie. The final Gold layer includes information about sleep sessions, wake events, and model factors.

Adding Machine Learning

Once the sleep data was structured, I built a machine-learning workflow using physiological and movement features. I used MLflow to track the model and its performance. The resulting model achieved approximately 51.34 percent balanced accuracy. I also incorporated feature importance into the Gold analytics layer so the application could show which signals were contributing to the model's predictions.

The SleepLens Application

I used Databricks Apps to turn the pipeline and model results into an interactive application. The app shows metrics such as Sleep Quality Score, Sleep Duration, and Sleep Efficiency, and it also lets users compare different nights. In the current prototype, one night had a Sleep Quality Score of 76.5 while another had a score of 72.7, which makes it easier to see differences in sleep duration, efficiency, wake events, deep sleep, and REM sleep.

Genie at the Core of SleepLens

The main goal of SleepLens is not just to show sleep data, but to help users understand it. I created the Sleep Analysis and Factors Genie Agent and connected it to the SleepLens application. Genie has access to the Gold analytics tables, including gold_sleep_sessions, gold_wake_events, and gold_model_factors. This lets users ask questions such as which night had better sleep quality and what factors likely contributed to the difference.

Why Genie Made a Difference

Without Genie, SleepLens can tell someone their Sleep Quality Score. With Genie, the application can help them investigate why that score was higher or lower. Instead of requiring users to understand SQL, schemas, tables, feature importance, or model outputs, they can simply ask a question and receive an explanation grounded in the SleepLens data.

Why Databricks Was Useful for SleepLens

Databricks allowed the entire project to live within one ecosystem. I was able to move from raw multimodal wearable data to data engineering, Bronze, Silver, and Gold analytics, machine learning, MLflow tracking, model explainability, an interactive app, and conversational analytics with Genie. This made it possible to build the data pipeline, model, application, and conversational layer as one connected system.

Architecture

The final SleepLens architecture starts with wearable heart-rate and accelerometer data paired with EEG-derived sleep labels. That data moves through the Bronze, Silver, and Gold layers, then into the machine-learning and MLflow workflow, then into the Databricks App, and finally into the SleepLens dashboard and Genie Agent for natural-language sleep insights.

What I Learned

The biggest lesson from building SleepLens was that an AI application is much more than the model itself. The data needs to be cleaned and structured correctly, the model needs to be evaluated, the results need to be understandable, and the application needs to make those results accessible. Building SleepLens showed me how data engineering, machine learning, application development, and conversational AI can work together as one system.

What's Next for SleepLens?

SleepLens is currently a prototype and is not intended to provide medical diagnoses or medical advice. In the future, I would like to explore streaming data from real wearable or edge devices, adding more physiological sensors, replacing simulated environmental context with real temperature and sound sensors, expanding the dataset across more users and nights, improving personalization, exploring on-device inference, and giving Genie access to longer-term sleep trends.

My longer-term vision is for SleepLens to become an intelligent interface between wearable sensor data and human understanding.

Main SleepLens Page

Ingested Raw Data

Medallion Architecture

ML model performance

Genie's Impact

🧪 MAD DATA LAB: Wonderful. Something Is Wrong.

___angel___ — Tue, 01 Sep 2026 02:15:14 GMT

Hi, I’m Ángel. I build data systems, break them more often than I would like to admit, and write about what I learn at Angelic Articles (my new own website for articles, previously on Medium).

What I enjoy most is usually not getting the answer. It is figuring out why the answer is true.

That is why the Databricks Genie-Powered App Challenge caught my attention — especially "Track B: Creative Thinking".

Most AI/BI experiences follow a familiar pattern:

Ask a question ➔ get an answer.

Useful? 🤔 Absolutely. But for a creative challenge, I wanted to turn that interaction around.

What if Genie did not simply wait for me to ask the right question?
What if the unexpected number itself became the problem to solve?

That question became MAD DATA LAB: a small analytics game where Dr. Genie forms hypotheses, chooses analytical experiments, follows the evidence and only reaches a conclusion when the numbers actually support it.

I decided to submit it because it combines two things I care about: making data exploration more approachable, and treating analytical answers as something to prove, not merely generate.

Because sometimes the interesting part is not the answer. It's proving it.

🔬 Welcome to MAD DATA LAB

MAD DATA LAB is built around one simple idea: Something in the data is wrong, and you have to prove why.

It's designed for analysts, data engineers, data scientists, BI users — and, more generally, anyone who has ever stared at a KPI and thought: that cannot be right.

Each investigation is a Case. A Case starts with an unexpected result, several plausible explanations and no conclusion that the player is expected to accept on faith.

The challenge demo is "Case #042 — The Missing €6.8M":

Expected €125.0M Observed €118.2M Deviation -€6.8M

The data is synthetic and deterministic, so the mystery is reproducible. The analytical evidence, however, is not just story text: the investigation is built around queryable evidence in Databricks.

Dr. Genie begins with competing hypotheses. Perhaps the source values changed. Perhaps the formula changed. Perhaps a suspicious data-quality signal is responsible.

The player predicts, inspects evidence and can ask for help. But the player does not manually choose the analytical route.

The app asks Genie what should be investigated next.

Genie might first decompose the deviation and find that one component, V2, contributes -€5.9M — roughly 87% of the total anomaly. The next Experiment can compare V2 across source snapshots and find:

23 modified records -€5.2M 2 removed records -€0.8M 5 added records +€0.1M -------------------------------- Net source impact -€5.9M

From there, the investigation can move down to individual records, lineage, formula validation, data-quality materiality and final reconciliation.

The loop is deliberately simple:

Case ➔ Hypotheses ➔ Experiment ➔ Evidence ➔ Update ➔ Repeat ➔ Scientific Verdict

And there is no game-over screen for guessing wrong.

The whole point is to watch your first theory collide with the evidence. Because that is usually where analytics gets interesting.

🧞 Genie Is Not the Hint Button

One thing I wanted to avoid was building a normal application and then attaching an AI assistant to the side.

In MAD DATA LAB, Genie is not there to explain a chart after the interesting work is finished.

Genie is part of the investigation loop.

At each step, the application gives the Genie Agent the current visible evidence and a server-controlled set of Experiments that are valid at that point. Genie evaluates the evidence, updates the hypotheses and chooses the next analytical move.

The distinction matters:

The application controls what is possible. Genie decides what makes sense next.

The same rule applies to visualisation. Genie can select from approved analytical Instruments — a deviation decomposer, snapshot comparison, evidence table, DQ panel, lineage view, reconciliation view and others — but it cannot invent arbitrary UI or executable code at runtime.

Users can also ask Dr. Genie questions such as:

Which component explains most of the deviation?
What changed between these two snapshots?
Is this data-quality warning actually large enough to explain the anomaly?
Which records contributed most?
Where did this value come from?

Under the hood, the data flow is intentionally small:

MAD DATA LAB ↓ Investigation / state orchestration ↓ Genie Agent ↓ Curated Unity Catalog evidence ↓ Databricks SQL

The application owns state, validation, scoring and safe rendering. Genie works against a curated analytical surface rather than unrestricted data.

There is also one boundary I care about a lot: Genie does not get the answer key.

The project has a private CASE_TRUTH oracle used for generation and automated validation. It's deliberately excluded from the Genie-facing data model. Dr. Genie has to reach the conclusion from the same visible evidence the investigation exposes.

Remove Genie, and MAD DATA LAB does not become the same game without a chatbot.

It becomes a scripted dashboard wearing a laboratory coat.

💥 Then Reality Entered the Laboratory

Of course, the first version of the idea was not the final one.

A few things looked excellent on paper and considerably less excellent five minutes later.

The beautiful board that was not actually useful

My first game-board concept looked wonderfully mad-scientist-ish. It also spent far too much of the screen being decorative.

Very pretty. Very atmospheric. Not particularly useful for investigating data.

So I killed it. The laboratory stayed, but the evidence had to become the protagonist.

Apparently, naming things is hard

At one point I was calling the whole investigation an Experiment… while also calling each individual analytical test an Experiment. That survived until I tried explaining the game out loud.

The vocabulary became:

Case ➔ Investigation ➔ Experiment ➔ Evidence ➔ Verdict

Much better.

The scary warning that explained almost nothing

Case #042 contains a genuine data-quality signal inside the synthetic Case data:

5 overlapping business keys. Estimated overlapping impact: -€0.3M.

The anomaly is -€6.8M. It's exactly the kind of thing humans — and AI — can jump on because it looks suspicious. But suspicious is not the same as material. And because that -€0.3M overlaps evidence already represented elsewhere, it must not be counted twice.

That became one of the central rules of the game: A warning is evidence. It is not automatically a cause.

Giving Genie freedom… but not a chainsaw 🪚

My first instinct was basically: let the AI decide everything.

That sounds elegant until the AI decides it would quite like an analytical instrument your application has never heard of.

The correction was simple: give Genie freedom inside explicit boundaries.

Enough freedom to investigate. Not enough freedom to invent a Quantum Revenue Microscope at runtime.

🧯 What Worked, What Didn’t, and What Surprised Me

A few design decisions made the project much stronger.

Deterministic Cases worked. If the underlying mystery changes every time, it becomes difficult to tell whether Genie improved or the crime scene simply moved.

Curated evidence worked better than “give it everything.” The more clearly the tables, fields, semantics and examples described the analytical world, the less I needed to compensate with instructions.

Reconciliation became non-negotiable. If an explanation claims to account for a €6.8M anomaly, the evidence eventually needs to add up to €6.8M. If it does not, the investigation is not finished.

What I moved away from was equally useful: giant instruction prompts trying to anticipate every situation, free-form chat as the primary game mechanic, arbitrary AI-generated UI, and adding more Cases before Case #042 was trustworthy.

There is also a deliberately boring reliability layer. Model output is validated before it can control the game, and the design includes a deterministic SQL fallback for evidence retrieval when Genie has already selected a valid Experiment but the normal query-result path fails. Offline fixtures are for development or a genuine platform outage — not the normal challenge experience.

The biggest surprise, though, was that the interesting moments were not when Genie instantly found the correct answer. They were when new evidence forced the investigation to change direction.

That felt much closer to real analysis than I expected.

🧠 What Genie Taught Me About Genie

Building MAD DATA LAB changed a few of my assumptions about analytical AI.

Better context beats bigger prompts

In this project, clearer semantics, curated data and tested examples were more valuable than increasingly heroic prompt engineering.

That matches the way Genie is designed to be curated: good metadata, business context, example SQL and realistic benchmark questions matter.

Freedom needs boundaries

Genie is most useful here when it can decide what to investigate, while the application defines what is legal and renderable.

A fully scripted flow is not very intelligent. An unconstrained AI application is not very predictable. The useful space is in between.

Test the evidence, not the prose

I stopped caring whether Genie says "Aha!" or “Interesting.”

What matters is whether it chose a sensible Experiment, retrieved the correct evidence, respected the numbers and reached a conclusion that reconciles.

The wording can vary. The facts cannot.

This is also why the test strategy focuses on numeric results, valid Experiment choices, hypothesis status and reconciliation rather than exact sentences.

“I don’t know yet” is a perfectly good answer

One of the easiest mistakes with AI is expecting a confident conclusion every time. But sometimes the evidence is simply not sufficient yet.

MAD DATA LAB uses explicit states such as POSSIBLE, SUPPORTED, CONFIRMED and RULED OUT so that “plausible” does not quietly turn into “proven.”

The interesting part is changing your mind

The moment that made the concept click for me was not Genie finding the right answer.

It was the investigation updating a hypothesis because new evidence contradicted the previous direction.

That is much closer to useful analysis: observe, hypothesize, test, revise.
Not: ask once, sound confident, move on.

🧪 The Scientific Verdict

I joined this challenge wondering whether Genie could do something more interesting than wait for a question.

MAD DATA LAB became my answer. It's playful on the surface, but underneath it is built around a serious analytical idea:

Do not trust the first explanation just because it sounds plausible. Test it. Quantify it. Reconcile it.

That is also what I ended up learning about Genie. The best experience did not come from asking it to sound smarter. It came from giving it better evidence, clearer boundaries and enough freedom to revise the investigation when the data changed the story.

So, after all the hypotheses, experiments, false leads and suspicious numbers, the final verdict is probably the simplest one: We did not ask for an answer. We ran an investigation.

🔧 Want the Technical Deep Dive?

This article intentionally focused on the idea, the experience, the mistakes and what I learned while building MAD DATA LAB.

For the less sensible amount of technical detail — architecture, Genie conversation orchestration, the closed Experiment/Instrument protocol, deterministic Case generation, curated evidence, hidden ground truth, SQL reconciliation, testing strategy, Genie benchmarks, security boundaries and failure handling — And someday, when I feel like documenting the less sensible amount of technical detail behind all this, I’ll probably turn it into a companion engineering deep dive on Angelic Articles.

Links

💻 Source: https://github.com/SauronShepherd/mad-data-lab

🎥 Video: https://youtu.be/RzGWBMzAXVc

BI Rationalization Genie: Turning Report Sprawl into Conversational Decisions with Databricks Genie

gbhogle1789 — Tue, 01 Sep 2026 01:49:52 GMT

Challenge Track: Track A — Real-World Problem Solver
Built on: Databricks Free Edition

Large enterprises rarely have a reporting problem because they lack reports. More often, they have too many — but the critical problem is determining which reports are genuinely needed, which can be consolidated or retired, and which require careful review before any action is taken.

As BI estates grow across platforms such as OBIEE, Tableau, Cognos, and analytical cubes, reports accumulate over time. Some are duplicates or slight variants. Some were created for one-time needs. Others are no longer actively used.

But rationalization is not as simple as finding an unused report and deleting it.

A low-usage report may still support an executive process, regulatory requirement, customer commitment, financial dependency, or another business-critical function.

That creates the real question:

How can business and technical teams understand what should be kept, consolidated, reviewed, or retired — without repeatedly going back to spreadsheets, SQL, and manual analysis?

This real enterprise problem inspired my submission for the Databricks Genie-Powered App Challenge: BI Rationalization Genie.

For the challenge, I built a Databricks App with a Genie Agent at the center of the analytical experience.

The goal was not just to replace SQL with chat.

It was to let different users investigate the same governed BI estate naturally — starting with a high-level question, drilling into why a decision was made, and then looking for broader patterns without having to predefine every analytical path.

The application is designed for BI leaders and executives, business and report owners, and technical teams responsible for rationalization and modernization.

The Data Behind the App

For the challenge, the inventory contains 300 reports across four BI platforms: OBIEE, Tableau, Cognos, and AAS.

The curated inventory contains both report metadata and rationalization evidence, including fields such as:

source platform
business criticality
decision rule
proposed action
final status
mandatory category
consolidation target
usage and other report-level indicators

The outcome of the upstream rationalization process is one of four dispositions:

KEEP | CONSOLIDATE | RETIRE | REVIEW

A short note on the upstream process

The final inventory does require processing before Genie uses it.

At a high level, the rationalization process combines BI metadata and usage signals, evaluates rationalization indicators such as report duplication, inactivity, one-time reporting patterns, hardcoded logic, filter variants, and column variants, and then applies business-criticality safeguards.

I intentionally kept these governed business rules outside Genie.

The rationalization process creates evidence and recommendation. Genie lets users investigate that evidence conversationally.

This separation was important to me. I did not want an AI agent to invent a retirement decision that should be governed by explicit business rules.

Architecture and Data Flow

The application follows a simple flow:

The curated inventory provides the analytical foundation.

The Genie Agent contains the business context needed to interpret questions about criticality, rationalization rules, proposed actions, review status, and other inventory concepts.

The Databricks App provides the user-facing experience for asking those questions and continuing the conversation.

One Interface, Different Questions

One thing I like about this pattern is that I did not need to design a different dashboard for every audience.

The interface stays the same.

The question changes.

An executive or BI leader might ask:

Which platforms should we prioritize for rationalization?

A business or report owner might ask:

Why does this critical report require REVIEW instead of being retired?

A data or BI engineer might ask:

Which decision rule triggered the recommendation, and what is the consolidation target?

The value is not role-specific screens.

It is allowing each user to follow the analytical path that matters to them using the same governed data.

Why Genie Is at the Core

This was the most important question I asked myself while building the app:

If Genie disappeared, what would change?

The rationalization recommendations would still exist in the inventory.

But the main experience would fundamentally change.

Users would return to predefined dashboards, spreadsheet filters, SQL queries, or requests to technical teams whenever they wanted to explore the estate in a way that had not already been designed for them.

The clearest way to demonstrate the value of Genie is through the different questions users can ask from the same governed BI inventory.

Executive View — Start with the Estate

An executive or BI leader can begin with a high-level question:

Q1. What does the overall BI estate look like across KEEP, RETIRE, CONSOLIDATE, and REVIEW, broken down by source platform?

Genie summarizes the estate across platforms and rationalization outcomes, giving leadership a quick view of where reports are being retained, consolidated, retired, or held for review.

From there, the user can move from visibility to prioritization:

Q2. Which source platforms show the largest rationalization opportunity based on retirement, consolidation, and review patterns? Explain using only the available data.

Instead of requiring a predefined dashboard for every possible comparison, Genie can analyze the governed inventory across source platform, proposed action, final status, criticality, and rationalization patterns.

This gives leadership a way to explore the estate based on the question they need answered at that moment.

Technical View — Investigate the Decision

Technical users can take the conversation deeper.

I asked:

Which critical reports were recommended for retirement or consolidation but require manual review instead?

Genie identified the affected reports and explained the relationship between the original rationalization recommendation, business criticality, and the final REVIEW status.

The next turn demonstrates an important part of the conversational experience.

I did not repeat the population report or rebuild the previous filters.

I simply asked:

Of those, explain which ones were originally consolidation candidates, what decision rule triggered each recommendation, what target report they consolidate into, and why criticality changed the final status to REVIEW.

Genie understood what “those” referred to from the previous turn.

It continued from the existing conversation, narrowed the analysis to the consolidation candidates, connected each recommendation to its rationalization rule and target report, and explained why the criticality safeguard prevented automatic consolidation.

The user does not need to know fields such as critical_flag, decision_rules, target_report_id, proposed_action, or final_status.

They can continue the investigation using business language.

From Investigation to Engineering Prioritization

The analytical path can then shift again.

A technical team preparing for modernization can ask:

Based on the rationalization patterns, which platforms and decision-rule categories should the engineering team investigate first for migration, consolidation, or retirement? Use only measurable evidence from the available data and do not assume timelines, effort, or business impact.

This question moves beyond identifying individual reports.

Genie can compare measurable patterns across the estate—such as report volume, rationalization actions, decision rules, criticality, and action readiness—to help engineering teams determine where deeper investigation should begin.

Importantly, Genie is instructed not to invent implementation timelines, effort estimates, or business impact when those facts are not present in the data.

This is the experience I wanted to create.

The same governed inventory supports several levels of investigation:

What does the estate look like?

→ Where are the largest rationalization opportunities?

→ Which reports require attention and why?

→ What evidence produced those decisions?

→ Where should engineering investigate next?

The analytical path is not hard-coded into a collection of separate dashboards.

It emerges from the user's questions and can continue as the investigation evolves.

That is where Genie becomes load-bearing in this application.

What Else Can Users Ask?

The same inventory also supports questions such as:

Which reports appear to be one-time or ad-hoc reports?
Which reports contain hardcoded dates?
Which reports can be consolidated, and what are their target reports?
How do rationalization outcomes differ across BI platforms?
Which critical reports prevent an otherwise automated rationalization action?

The user does not need to understand the schema before beginning the analysis.

They start with the business question.

Building for Accuracy Was a Bigger Part of the Work Than I Expected

Connecting Genie to the inventory was not the difficult part.

Getting consistently correct answers was where I spent more time.

I created benchmark questions around important rationalization scenarios and compared Genie's responses with the expected results.

After iterating on the semantic context and testing the generated queries, I completed a clean benchmark run in which all five benchmark questions passed.

The benchmark set covered important rationalization scenarios such as critical-report safeguards, retirement recommendations, consolidation candidates, one-time reports, and hardcoded-date patterns, giving me a repeatable way to validate Genie’s accuracy against known expected results.

One debugging habit became particularly useful:

When Genie gave me an incorrect answer, I stopped immediately adding more instructions.

Instead, I checked:

Is the underlying data correct?
What SQL did Genie generate?
Is the business terminology clear?
Does Genie have the semantic context needed to interpret the question?
Is my expected benchmark answer itself defined correctly?

That process led to one of my most useful learnings from the challenge.

Instructions and evaluation notes are not the same thing

At first, when an evaluation failed, the natural reaction was to think: I need another instruction.

That is not always the right fix.

I started using a much simpler distinction:

Instructions help guide Genie. Evaluation notes help evaluate Genie.

If Genie repeatedly needs business context to interpret future questions correctly, that belongs in its instructions or semantic knowledge.

If the issue is defining what the benchmark judge should consider a correct response, that belongs in the evaluation note.

This small distinction helped me avoid continuously adding instructions just to make an evaluation pass.

I also learned that inspecting the generated SQL can often tell you more about an accuracy problem than adding another paragraph of prompting.

Another Lesson: Deployed Does Not Always Mean Accessible

The app integration gave me another very practical lesson.

At one point, the Databricks App was successfully deployed and running, but that did not automatically mean the complete Genie experience was accessible.

The application identity also needed the appropriate access to the Genie Agent and the required underlying resources.

That changed how I thought about testing the solution:

Test the Genie Agent. Test the app. Then test the authorization path between them.

For an enterprise implementation, the same principle extends to governance.

The conversational interface should make analytics easier to use, but it should not create a shortcut around the controls protecting the underlying data. Access to the app, Genie, and the governed inventory should continue to follow the appropriate Databricks authorization and Unity Catalog controls.

What I Took Away From the Challenge

Three lessons stand out for me.

Do not treat every Genie accuracy problem as a prompting problem.
Look at the data, generated SQL, semantics, and expected answer first.
Conversation changes the value of analytics.
The useful part is not only asking the first natural-language question. It is being able to continue from that answer, change analytical direction, and explore something that was not predefined in the UI.
A working Genie Agent and a working Genie-powered application are two different milestones.
Integration, authorization, conversation handling, and application behavior need their own testing.

Closing Thought

I started this project with a report-rationalization problem.

The upstream process can already calculate governed recommendations such as KEEP, CONSOLIDATE, RETIRE, and REVIEW.

But a recommendation sitting in a table is not the same as making that information easy for people to investigate.

That is the role Genie plays in this application.

A BI leader can start at the estate level.

A report owner can ask why a particular decision was made.

A technical user can drill into the rule and supporting evidence.

And they can move between those questions through a conversation rather than waiting for another query, spreadsheet, or dashboard to be built.

The rationalization process creates evidence. Genie turns that evidence into an investigation.

#Databricks #DatabricksGenie #DatabricksApps #DataAI #BusinessIntelligence #DataModernization #GeniePoweredAppChallenge

Crux — Construction Intelligence Powered by Databricks Genie

Areddy10 — Tue, 01 Sep 2026 01:13:32 GMT

Databricks Community Genie-Powered App Challenge. Track A: Real-World Problem Solver.

Managing a construction portfolio means constantly answering questions across schedule, cost, tasks, contractors, inspections, change orders, and field reports. The problem is that the answer to a single question can be split across structured project data and pages of project documentation.

Crux brings those two worlds together.

A user asks a question in plain English. Databricks Genie establishes the authoritative structured picture: project risk, schedule variance, forecast cost, blockers, overdue work, contractor performance, and portfolio comparisons. When the question also needs context, Crux retrieves supporting project evidence that explains what is happening and why.

Genie discovers and quantifies the problem. Documentary intelligence explains and substantiates it. Crux turns both into a decision.

The idea, and who it is for

Consider a project manager trying to understand why a project needs attention.

The schedule may show a delay. The cost data may show a forecast overrun. Tasks may reveal overdue work. An inspection record may identify a blocker. A field report may finally explain the operational reason behind it.

Getting the complete picture normally means moving between different systems, reports, and datasets or relying on someone who knows how to query them.

Crux is built for project managers, operations teams, and construction leaders who want to ask those questions directly:

Which projects are at high risk?
Why is Stellar Heights at risk?
What inspection is blocking it?
What about its forecast variance?

The user does not need to know the schema, write SQL, or manually search through project documents.

One design decision became fundamental while building Crux:

Not every source should have equal authority.

Genie owns the structured business facts. Project documents provide supporting explanation and evidence.

One project, end to end

Take Stellar Heights.

Crux includes a Projects workspace where the portfolio can be explored before moving into a deeper investigation. Selecting Stellar Heights takes the user into the Intelligence workspace with the project context prepared:

What should I know about Stellar Heights?

Genie establishes the authoritative structured state:

High Risk
18 days behind schedule
8.10% forecast cost variance
$3,928,500 forecast overrun
7 overdue tasks
1 active blocker

Those numbers answer what is happening.

But construction decisions often require the next question: why?

For the same inquiry, Crux retrieves relevant documentary evidence describing issues affecting Stellar Heights, including an HVAC fabrication backlog and staffing constraints, the INS-2003 electrical inspection issue, and its downstream impact on drywall work.

The two intelligence sources contribute to the same experience without competing for authority.

Genie establishes the measurable project state. Documentary intelligence explains the conditions behind it.

The answer is also inspectable.

Documentary claims include citations back to the retrieved project evidence, with source metadata such as the document and relevant location in the source.

The structured analysis is transparent too. Crux exposes the SQL generated by Genie alongside the result, so the user can inspect how the structured answer was produced instead of treating it as a black box.

The investigation can then continue naturally.

Ask:

What inspection is blocking it?

Crux understands that the conversation still refers to Stellar Heights and identifies INS-2003, together with the relevant supporting evidence.

Then ask:

What about its forecast variance?

Genie continues with the existing project context and returns the forecast variance without requiring Stellar Heights to be repeated.

The previous turns remain available in the same intelligence thread, including structured results, documentary evidence, citations, and generated SQL.

Genie at the core

Remove Genie from Crux and the core product no longer works.

The documentary layer could still retrieve project reports, but Crux could no longer establish authoritative project risk, schedule variance, forecast cost variance, overdue work, blockers, contractor performance, rankings, or portfolio-level metrics.

Genie is load-bearing in several ways.

Genie owns the structured truth.
Project and portfolio metrics come from Genie over governed Databricks data. Crux does not independently recalculate those metrics after Genie responds.
Genie turns business questions into structured analysis.
The user asks in natural language. Genie determines the relevant data, generates the SQL, executes the analysis, and returns the structured result. Crux preserves and exposes the generated SQL.
Genie holds the structured conversation.
Follow-up questions continue the same Genie conversation. A question such as “What about its forecast variance?” can therefore build on the previous discussion instead of starting from zero.
The Genie Space carries the construction semantics.
Instructions, business definitions, relationships, and verified examples guide how concepts such as project risk, schedule variance, blockers, change orders, and contractor performance should be interpreted.
Genie stays authoritative when documents enter the answer.
A project report may explain why something happened, but it cannot redefine a Genie-owned KPI. Documentary evidence can support the metric; it does not become the source of truth for it.
Crux decides when Genie is required.
A supervisor routes each question into one of three paths: GENIE_ONLY, RAG_ONLY, or GENIE_PLUS_RAG. Structured business questions go through Genie, documentary questions can use retrieval, and broader project questions can invoke both.

That responsibility boundary became the core of the architecture:

Genie tells Crux what is true in the structured portfolio. Documentary intelligence helps explain why.

What you can ask it

Crux is not built around a fixed set of dashboard filters or pre-written queries.

Users can ask questions such as:

Which projects are at high risk?
Which of those is the most delayed?
What should I know about Stellar Heights?
What inspection is blocking it?
What about its forecast variance?
Which contractor has the most task slippage?

The user does not need to know which table contains the answer, how the tables join, which document mentions the issue, or what SQL should be written.

For a portfolio question, Genie performs the structured analysis.

For a documentary question, Crux retrieves the relevant project evidence.

For questions requiring both, the two paths are combined while maintaining their authority boundary.

Architecture

Crux runs as a Databricks App.

A Databricks-served supervisor receives the user's question and chooses between structured intelligence, documentary intelligence, or both.

The structured path uses Databricks Genie over governed Unity Catalog data for projects, project costs, tasks, contractors, inspections, and change orders through a SQL warehouse.

The documentary path begins with project files stored in Unity Catalog Volumes. Documents are parsed and deterministically chunked into governed tables, then indexed with Databricks Vector Search using hybrid retrieval.

For mixed questions, Genie establishes the authoritative structured state while the retrieval pipeline supplies supporting documentary evidence. Crux then combines the two deterministically so that the evidence layer cannot overwrite Genie-authoritative metrics.

The application also preserves the active Genie conversation and bounded project context for multi-turn questions.

What I learned building it

Authority was harder than routing.
Deciding whether to call Genie or retrieval was only part of the problem. Once both systems contribute to an answer, defining which system is allowed to state which facts becomes much more important.
More model calls did not mean a better system.
An earlier architecture included additional model calls for documentary planning and final synthesis. Evaluation showed they were unnecessary. Removing them produced a simpler mixed path while keeping Genie authoritative.
RAG quality starts before retrieval.
Document identity, parsing, chunking, versioning, serving eligibility, indexing, and citation metadata all had to be deterministic before the retrieval layer became dependable.
A chat interface does not automatically create a conversation.
Multi-turn behavior required preserving the real Genie conversation and resolved project context. After a reset, an ambiguous question such as “What about its forecast variance?” correctly loses that context.
Transparency belongs in the product.
Generated SQL and documentary citations are visible to the user. Natural language makes Crux accessible; inspectability makes the answer useful for decision-making.

The result

Construction intelligence is not only about finding a number.

A project leader needs to know which project needs attention, how significant the problem is, what is causing it, and what evidence supports that conclusion.

Databricks Genie provides the structured intelligence core that makes those questions accessible through natural language.

Crux builds the decision workflow around it.

Genie discovers and quantifies the problem.

Documentary intelligence explains and substantiates it.

Crux turns both into a decision.

CRUX - Construction Intelligence, powered by Databricks Genie

OpsPulse- From Operational Signals to Data-Backed Answers with AIBI Genie

kulkarnigauri3 — Tue, 01 Sep 2026 00:34:03 GMT

OpsPulse- From Operational Signals to Data-Backed Answers with AIBI Genie

Operations teams usually do not have a shortage of data. They have dashboards, reports, KPIs, spreadsheets, and weekly reviews.

The harder part often starts after someone notices that a metric has moved.

Why is backlog increasing?
Is the issue higher inbound volume or lower processing capacity?
Which vendor is actually deteriorating?
Is an increase meaningful, or just normal variation?

I built OpsPulse for that part of the analytics workflow.

OpsPulse is a Genie-powered operations intelligence app built on Databricks Free Edition. It allows an operations manager or analyst to investigate backlog, SLA attainment, throughput, defect rates, downtime, vendors, sites, and processes using natural-language questions.

My goal was not to build another dashboard or simply put a chat interface on top of a table. I wanted to see how far I could take a governed operational analytics experience where the user can move from:

Detect → Diagnose → Investigate

and where the system also knows when the available data is not sufficient to answer a question.

The use case

Imagine an operations manager opens the application and asks:

What operational issues need attention in the latest available data?

Instead of requiring the manager to open multiple dashboards and compare several KPIs manually, OpsPulse evaluates recent operational performance against an appropriate historical baseline.

It can surface issues such as:

backlog growth
SLA deterioration
declining throughput
increasing defect rates
rising downtime

The user can then continue the investigation naturally.

For example:

Why did East Hub backlog increase after August 18?

In the test data, OpsPulse found that average daily inbound volume had increased to approximately 2,742 units, while average daily processed volume reached only about 2,288 units.

That created an average processing gap of approximately 454 units per day.

At the same time, average daily downtime increased from approximately 226 minutes before August 18 to more than 1,029 minutes after August 18.

This gives the user more than a statement that "backlog increased." It provides the operational context needed to understand what changed.

Who I designed OpsPulse for

I designed the app primarily for:

Operations managers
Business analysts
Data analysts
Supply chain and logistics teams
BI teams supporting operational decision-making

These users may understand their business very well without necessarily wanting to write SQL every time they have a follow-up question.

The value of Genie in this scenario is the ability to keep the investigation moving.

A user can start with a broad question, identify an exception, and immediately ask a more specific question without switching tools or rebuilding an analysis.

Architecture

The application uses a relatively simple architecture, but I spent quite a bit of time on the semantic and analytical layer behind Genie.

Synthetic Operations Data
        ↓
Delta Table
operations_daily
        ↓
Curated KPI View
operations_kpi_view
        ↓
Unity Catalog Metadata
+ Governed KPI Definitions
        ↓
AI/BI Genie Agent
        ↓
Databricks AppKit
        ↓
OpsPulse

The main components are:

Databricks Free Edition
Databricks SQL
Unity Catalog
Delta tables / views
AI/BI Genie
Databricks Apps with AppKit

I used a synthetic operations dataset for this project so that the behavior of the application could be tested against known scenarios.

The dataset contains 1,080 operational records covering 30 days, with combinations of:

4 sites
3 vendors
3 operational processes

The data includes measures such as inbound units, processed units, backlog, SLA-eligible units, SLA-met units, defects, labor hours, downtime, and processing time.

I deliberately introduced several patterns into the data, including an East Hub backlog increase, vendor SLA deterioration, a throughput improvement at another site, and a temporary defect-rate increase.

That made it possible to test whether Genie could actually identify and explain the patterns I knew were present.

Building a governed KPI layer

One thing I wanted to avoid was allowing every question to produce a slightly different KPI calculation.

For example, SLA attainment should not be calculated by simply averaging percentages from individual rows.

For aggregated analysis, I defined SLA attainment as:

Total SLA-met units / Total SLA-eligible units

I used the same approach for other measures:

Throughput

Total processed units / Total labor hours

Defect rate

Total defect units / Total processed units

Net backlog change

Sum of ending backlog minus starting backlog

Processing gap

Inbound units minus processed units

These measures were defined explicitly for Genie so that the same business definitions would be used across different questions.

I also added descriptions and metadata to the Unity Catalog view so fields such as labor_hours, processing_gap_units, and backlog_change_units had clear business meaning.

What users can ask

A few examples that worked well during testing were:

What operational issues need attention in the latest available data?

Why did East Hub backlog increase after August 18?

Which vendor experienced the largest decline in SLA attainment after August 16?

Where did defect rates increase abnormally?

Is East Hub's backlog problem primarily caused by higher inbound volume or weaker processing performance?

These questions demonstrate different parts of the workflow.

Some are broad exception-detection questions. Others are diagnostic questions that require Genie to compare periods, calculate weighted KPIs, or break performance down by site, vendor, and process.

The most useful part of the project was actually the testing

The part I learned the most from was not getting Genie to answer more questions.

It was learning how to improve the quality of the answers and prevent unsupported conclusions.

1. Arbitrary KPI thresholds

During an early test, I asked:

What operational issues need attention?

The first approach introduced thresholds such as SLA below a certain percentage or downtime above a certain number of minutes.

The problem was that I had never defined those thresholds as business rules.

So I changed the analytical approach.

For broad operational questions, OpsPulse now compares the latest seven days against the preceding fourteen-day baseline for the same site, vendor, and process.

This made the analysis much more useful because the application looks for deterioration relative to recent normal performance rather than inventing universal thresholds.

2. Choosing the right baseline matters

I also tested a latest-day comparison against the previous seven days.

That initially looked reasonable, but it created another issue.

If an operational problem had already existed for several days, the recent baseline itself contained part of the deterioration. That could make the latest day look relatively normal and hide the issue.

I changed the logic to compare the most recent seven-day period against the preceding fourteen days.

That simple change significantly improved the exception detection.

It was a good reminder that the quality of AI-assisted analytics still depends heavily on the analytical design behind it.

3. Labor hours are not labor cost

One of my favorite tests was:

What can you tell me about labor cost across the sites?

The dataset contains labor_hours, but it does not contain hourly wages, labor rates, or any monetary cost field.

In an early test, the response started treating higher labor hours as an indication of higher labor cost.

That was not supported by the data.

I updated both the Genie instructions and the Unity Catalog metadata to make the distinction explicit:

Labor hours measure operational effort. They are not a financial cost measure.

After the change, OpsPulse correctly responded that labor cost could not be determined from the available dataset.

That was an important result for me because a useful analytical agent should not only know how to calculate something. It should also know when the data does not support the requested conclusion.

4. "Which site is performing best overall?"

Another interesting test was:

Which site is performing best overall?

Initially, Genie created an implicit ranking using SLA, throughput, defects, backlog, and downtime.

But I had never defined a composite performance score or assigned weights to those KPIs.

There is no analytically defensible reason to assume, for example, that SLA should automatically matter more than defect rate, or that throughput should receive a particular weight.

So I added a guardrail.

Without a governed scoring methodology, OpsPulse now compares the sites across the individual KPIs and explains that the "best" site depends on the business priority.

This was another small change, but an important one from a data governance perspective.

Genie at the core of the application

Genie is not an optional feature inside OpsPulse. It is the primary way the user interacts with the operational data.

The experience is designed around a sequence like this:

1. Detect

What needs attention?

2. Diagnose

Why is East Hub backlog increasing?

3. Investigate

Is inbound volume increasing faster than processing capacity?

The application uses Genie to translate those business questions into queries against the curated operational KPI layer and return the results in a form that an operations user can work with.

Without Genie, the user would lose the natural-language path from an operational question to the underlying data and follow-up analysis.

Building the Databricks App

I used the AppKit Genie template in Databricks Apps and connected the application directly to the OpsPulse Genie Agent.

I customized the application around the three-stage workflow:

Detect

Surface emerging operational exceptions across sites, vendors, and processes.

Diagnose

Compare recent performance with historical baselines and identify the metrics contributing to deterioration.

Investigate

Ask follow-up questions in natural language and explore the supporting operational data with Genie.

I intentionally kept the application itself simple.

Most of the work went into the data model, KPI definitions, business metadata, query patterns, testing, and guardrails behind the interface.

What I learned

This project changed the way I think about conversational analytics.

Connecting an AI interface to data is relatively straightforward.

Making the answers consistently useful is much more interesting.

I found myself spending more time thinking about questions such as:

What does "latest" actually mean?
What comparison period is appropriate?
Should this KPI be averaged or weighted?
Is the system identifying a correlation or claiming a cause?
Is a threshold actually defined by the business?
Is the requested conclusion supported by the available fields?
Should the system rank something when no scoring methodology exists?

Those questions are not really AI questions.

They are analytics, semantic modeling, business logic, and governance questions.

And I think that is where tools like AI/BI Genie become particularly interesting for analysts.

Final thoughts

OpsPulse started as a fairly simple idea: let an operations manager ask questions about operational performance.

By the end of the project, the more interesting challenge became making those answers consistent, explainable, and grounded in the available data.

There is still plenty I would extend in a production version, including governed business targets, more sophisticated anomaly detection, additional historical context, alerting, and integration with real operational datasets.

For this challenge, though, I wanted to focus on one thing:

Can a user move naturally from noticing an operational signal to understanding what may be driving it, without losing the analytical discipline behind the answer?

OpsPulse is my attempt at that.

Built with

Demo: OpsPulse: From Operational Signals to Data-Backed Answers with AI/BI Genie

[https://drive.google.com/file/d/1MZR6scvsFNv7R8iCzHftYAO1V5KA9IIj/view?usp=sharing]

Genie SQL Quest: A Genie-Powered Arcade for Learning SQL

Chiku07 — Mon, 31 Aug 2026 22:31:07 GMT

🧞 What if learning SQL felt like a game?

SQL is one of the most valuable skills in data and AI, but most people still learn it from static tutorials or documentation. Chatbots can answer questions, but they rarely push learners to think through a problem, make mistakes, and try again. Genie SQL Quest changes that by turning SQL practice into a game — and it uses Databricks Genie as the intelligent coach behind every challenge.

I built this project for the Databricks Community Contest — Creative Thinking track to show that Genie can be more than a conversational analytics assistant. In Genie SQL Quest, Genie becomes a real-time tutor: it gives hints when you are stuck, explains why your answer is right or wrong, and makes the learning experience feel natural and conversational. The goal is to make SQL practice feel less like a lecture and more like a guided puzzle.

✨ What Genie SQL Quest does

Genie SQL Quest is a browser-based learning game where players solve natural-language SQL challenges presented as flashcards.

Curated challenge categories: Players progress through Basics, Filtering, Aggregations, Sorting / Top-N, Joins, and Date logic. Each category is designed to build a specific SQL muscle.
Progressive difficulty: Every category contains multiple difficulty levels. Players must solve easier cards to unlock harder ones, which keeps the pacing adaptive and rewarding.
Boss Rounds: Once players master a category, they face a harder Boss Round that combines multiple concepts and tests deeper understanding.
XP, streaks, and progress tracking: Correct answers earn XP, streaks encourage daily practice, and the progress panel shows how far the player has come across every category.
Interactive schema panel: Players can inspect tables, columns, primary/foreign keys, sample rows, and relationships at any time. This removes the friction of memorizing a schema and lets players focus on writing good SQL.
Instant feedback: Every submission is checked against a gold answer in the question bank, so players immediately know if they are correct and see the canonical query.

While the game mechanics keep players engaged, the real intelligence comes from Genie.

🧠 How Genie is at the center

Databricks Genie is not a side feature in this project — it is the core reason the app feels intelligent and conversational. The backend calls the Databricks Genie Conversational API to bring the flashcards to life.

Capability How Genie powers the experience

Hints on demand	When a player is stuck, Genie reads the current challenge and produces a short, context-aware hint. The hint points the player toward the right SQL concept without giving away the answer.
Personalized explanations	After a player submits an answer, Genie explains the key idea behind the correct SQL in one friendly sentence. The explanation can be tailored to whether the player's answer was correct or incorrect.
Boss-round narration	Genie can introduce or comment on Boss Round challenges, adding a narrative layer to the hardest part of the game.
Graceful fallback	If Genie is not configured, the app falls back to static hints and explanations stored in the question bank, so the game is never broken.

The integration is implemented through a lightweight Python client that streams responses from the Genie API via Server-Sent Events and extracts the assistant's message. This makes every Genie interaction fast, simple, and focused on the current challenge.

🏗️ Architecture

The application is split into three layers:

1. React + Vite Frontend

Category panel for picking challenge tracks and difficulty levels.
Flashcard play area with the prompt, SQL input, and submission flow.
Hint buttons that request Genie-powered guidance.
Progress panel showing XP, streaks, cards solved, and category completion.
Schema panel showing table definitions, columns, relationships, and sample rows.

2. FastAPI Backend

Serves the game API endpoints: drawing cards, submitting answers, requesting hints, and resetting progress.
Loads the question bank and schema from JSON files.
Evaluates player answers against canonical SQL.
Orchestrates calls to Databricks Genie for hints and explanations.

3. Databricks Genie

Provides natural-language hints when requested.
Explains SQL answers based on the player's attempt and the challenge prompt.
Acts as the conversational AI tutor that makes the app feel alive.

Key design decisions

Game-first, Genie-enhanced: The question bank provides deterministic scoring so feedback is instant and fair. Genie adds the conversational layer on top without slowing down challenge.
No ETL pipeline needed: The dataset is a curated JSON question bank, so the project is easy to run, extend, and deploy.
Resilient to missing credentials: If Genie is not configured, the app automatically uses static hints and explanations so development and demos are never blocked.
Databricks Apps ready: The app is packaged with an app.yaml and a single-entry run.py for straightforward deployment inside Databricks.

🛠️ Tech stack

Frontend: React, Vite, Tailwind CSS, Lucide icons
Backend: FastAPI (Python)
AI/BI layer: Databricks Genie Conversational API
Optional fallback: Databricks Model Serving endpoint (databricks-dbrx-instruct)
Deployment: Databricks Apps

🎯 Why Genie makes this work

Genie is the difference between a static SQL quiz and an adaptive learning experience. Without Genie, the app would only show pre-written hints. With Genie, the app becomes a tutor that understands the current challenge and the player's actual answer.

The three things that make Genie essential here are:

Context-aware guidance: Genie reads the exact challenge the player is looking at, so every hint is relevant and timely.
Personalized feedback: Genie can react to whether the player's answer was correct or incorrect, making the explanation more useful.
Conversational learning: Because Genie responds in natural language, players feel like they are learning from a coach rather than grading against an answer key.

That combination — structured, level-based challenges plus Genie's conversational intelligence — is what turns SQL practice into something players actually want to come back to.

Built for the Databricks Community Contest. Feedback, forks, and questions welcome!

LakeOps - A Lakehouse Operation Observability App

ad3ad3 — Mon, 31 Aug 2026 20:38:05 GMT

The Problem LakeOps Solves

Data teams focused on cloud-based analytics solutions often experience operational bottlenecks. Engineers spend countless hours writing SQL joins across system logs monitoring and reporting pipeline failures, while engineering managers struggle to get visible insights into compute costs and SLA performance. Some teams solve this through dashboards with static KPI reporting, but this often limits how much insight could be drawn from the data. LakeOps addresses this by providing an intelligent, natural-language interface for Lakehouse operation observability.

Target Audience and Dual-Persona Design

LakeOps is designed with a context-aware toggle to serve two distinct operational roles:

Data Engineers: Can ask highly technical debugging questions to retrieve specific error traces, execution times, and job logs for root-cause analysis.
Engineering Managers: Can request high-level summaries regarding DBU consumption, overall pipeline costs, and business impact.

Application architecture and data flow

LakeOps utilizes a secure architecture deployed directly within the Databricks Data Intelligence Platform. The end-to-end data flow is visualized below:

Data Layer: Following the Medallion architecture, the pipeline processes data through three layers, all stored in Delta Lake and strictly governed by Unity Catalog:

Bronze: Raw logs and system tables are used as the bronze layer. Since this project was developed using the Databricks Free Edition workspace, synthetic data was ingested into the tables in this layer.
Silver: Transforms and cleans the raw data from the bronze layer into parsed pipeline execution records and technical error traces as required.
Gold: Aggregates the silver data into final business-level metrics, such as total DBU consumption, costs, and SLA performance.

Authentication Flow: The Streamlit application is hosted as a serverless, containerized Databricks App. It utilizes On-Behalf-Of (OBO) authentication, meaning the app's Service Principal is entirely locked out of the data. Instead, it uses the logged-in user's OAuth token, ensuring all queries execute securely under their exact Unity Catalog privileges.

Logic & Processing with Genie Agent: When a user submits a prompt, the app injects the selected persona context (Data Engineer or Engineering Manager). The Databricks Genie Agent processes this combined context, acting as the semantic engine to translate the natural language request into optimized SQL against the permitted tables.

Presentation Layer: The app's custom logic intercepts the returned SQL and data arrays from the Databricks SDK. It then intelligently routes the output to render dynamic, interactive Plotly tabs (Bar, Line, Pie) or raw DataFrames based on the structure of the returned data.

What can users ask the Genie Agent

Since LakeOps is context-aware, queries are inherently tailored to the user's operational role. The application intelligently guides the agent to fetch the appropriate level of detail:

Engineering Managers can ask business-level, aggregate questions such as:

"What was our total DBU spend by pipeline this month?"
"Which pipeline experienced the most SLA breaches last week, and what was the associated compute cost?"

Data Engineers can dive directly into technical, root-cause investigations by asking:

"What is the success rate for all pipelines in the last month?"
"What is the average execution latency for the Gold aggregation job over the past seven days?"

How Genie powers the app's main experience

When a user submits a question, the Streamlit app concatenates the persona-specific instructions and forwards the prompt to Genie. Genie leverages its deep understanding of the underlying table metadata and Unity Catalog relationships to generate optimized SQL queries on the fly. LakeOps then extracts this SQL from the Genie API payload, executes it synchronously against a serverless SQL Warehouse, and uses a custom Python heuristic to dynamically render the results as interactive Plotly charts or raw data tables. Without Genie's real-time text-to-SQL translation and semantic awareness, this dynamic, self-serve observability experience would be impossible.

What I learned while building and testing the app

Building and deploying LakeOps end-to-end within the Databricks Data Intelligence Platform provided several key engineering takeaways:

Curating Genie Data Models for Accuracy: Grounding Genie in accurate metadata, logical table joins, and clear instructions is essential to prevent hallucinations. Curating focused tables across the Medallion layers ensured Genie consistently chose the correct tables depending on the context of the incoming question.
Benchmarking Genie Agent Queries: Conducted query benchmarking to establish baseline performance and response accuracy. This testing phase was vital to verify that Genie consistently translated natural language into the most optimized SQL, and to ensure acceptable latency when querying across the varying complexities of the Medallion layers.
Mastering On-Behalf-Of (OBO) Authentication: Implementing user authorization via the OBO model proved critical for enterprise security. It taught the importance of properly configuring OAuth scopes (genie, workspace, SQL) so the app can securely execute queries under the end-user's identity without relying on overly privileged Service Principals.
Building Resilient API Parsers: Handling SDK payload structures required robust dictionary-based parsing to extract text attachments and generated SQL statements dynamically across varying API responses.
End-to-End Serverless Integration: Deploying a containerized Streamlit Databricks App directly where the data lives highlighted the efficiency of the serverless compute plane. Coordinating the workflow—from the UI, through the Genie Agent, across the serverless SQL Warehouse, and back to dynamic Plotly visualizations—demonstrated how seamlessly Databricks components integrate into a unified operational tool.

Final Thoughts and Future Roadmap

While LakeOps successfully demonstrates the power of natural-language observability, the current application is a prototype. Time constraints limited the initial scope, but the following enhancements would significantly elevate both the app's capability and the overall user experience:

General User Experience Improvements

Automated Data Refresh Pipelines: Implementing a scheduled daily pipeline orchestration job to keep data continuously flowing from the Bronze to Gold layers, ensuring that LakeOps always surfaces fresh operational metrics from the previous workday.
Query Caching: Implementing caching mechanisms (such as Streamlit's @ST.cache_data) for repetitive, high-level SLA queries to ensure instant load times and minimize SQL Warehouse compute costs.
Proactive Daily Health Checks: Adding automated routines that execute critical SLA and cost queries upon launch, presenting users with a summarized morning health briefing before they even type a prompt.
Follow-Up Question Suggestions: Utilizing lightweight LLM calls to generate contextual, clickable follow-up questions beneath rendered charts, driving continuous data exploration.

RAG Integration for Autonomous Remediation To transition LakeOps from a purely diagnostic tool into an active remediation assistant, the architecture could be expanded with Retrieval-Augmented Generation (RAG):

Cross-Platform Documentation Search: Indexing official documentation for the core technologies in the team's data stack—such as Databricks and Azure Data Factory. If an engineer encounters an obscure pipeline exception, the RAG model could surface the exact syntax fix or known workaround without requiring them to leave the app.
Historical Long-Term Memory: Ingesting past Jira tickets, Slack support threads, and incident post-mortems into a Databricks Vector Search index. This would grant the application long-term memory, instantly providing engineers with historical context on recurring pipeline failures and their exact resolution paths.

This development was done using Google Gemini and Genie Code as coding assistant.

#Genie-PoweredAppChallenge