Databricks Community

okinashchuk · a week ago

Databricks Community Genie-Powered App Challenge · Track B, Creative Thinking

A dump of the customer database has appeared on a public forum. Four hundred people work at this company and any of them could have taken it. The access logs know what happened — three hundred thousand rows of queries, badge swipes, VPN sessions and file transfers — and nobody has read them.

You cannot read them either. There is no table browser, no dashboard, no filter panel. There is one way to reach the evidence, and it is to ask.

Watch the 4-minute demo:

*The app itself runs on Databricks Free Edition, where apps are authenticated against the owning workspace — so the recording is the way to see it work.

The idea

The organisers set one test: if Genie were removed, would the main experience change significantly?

Most apps answer that badly, because the obvious build is a dashboard with a chat box bolted to the side. Remove the chat box and the dashboard still works.

A murder mystery cannot cheat that test. If the archive stops answering, there is no game — not a degraded game, none at all. The player's question is the mechanic, so Genie isn't a feature of the app; it's the only interface it has.

That framing also solves a problem every Genie demo has. Natural-language BI demos are dull to watch because the questions are invented on the spot and nobody in the audience cares about the answer. In a mystery, the audience wants the answer before the question is finished.

Who it's for

Anyone about to curate a Genie space for the first time.

The failure modes in this build were not exotic. An instruction that described two network bands when the data had three. A table that recorded when leave started but not when it ended. An agent that reported "no records" about a table it had never queried. Every one of them produced a confident, fluent, completely wrong answer — and every one is the kind of thing you write on a Tuesday without noticing.

Reading that as advice is forgettable. Playing a case where a wrong label sends you after an innocent colleague is not. The game is a teaching device wearing a detective story, and it is aimed squarely at the person who is about to trust an agent because its answers sound right.

It also works for anyone who simply likes a puzzle. No SQL required — the entire game is played in plain English.

The constraint that makes it work

The interesting design problem is the obvious objection: won't Genie just tell you who did it?

It won't, and not primarily because I asked it not to. The answer isn't in the data.

There is no is_guilty column and no single-query path to a name. The culprit is identified by a conjunction that crosses three tables:

Reads against the customer table totalling exactly the stolen record count — assembled from 31 paginated queries, so no single query looks unusual.
Those queries ran under an account whose owner has no badge events that week and was on approved leave, overseas.
Yet the queries came from the office network — so somebody was physically at a desk. Badge records show exactly one person in the building during those hours.

Genie will perform every one of those steps when asked. It will not form the hypothesis. And "who took the database" has no answer in any schema, because guilt is not a column.

Three layers keep it honest, in descending order of how much I rely on them:

The answer isn't in the data, and the ground truth lives in server code outside the Genie space's table scope, so the agent cannot read it however it is asked.
The win condition is an accusation — name one person, cite three records — validated in application code. Even if the agent gave something away, the loop still runs through the player's judgement.
Space instructions cast the agent as a records archivist that refuses to speculate. Useful reinforcement; never the thing the design rests on, because instruction-following is best-effort.

Asked directly, it holds:

Who took the customer database?
I only retrieve records and cannot determine who "took" the customer database. If you want to know who queried or accessed the customer database ('main.crm.customers_pii') during a specific period, please specify the date range or type of access you are interested in.

What Genie is actually doing here

Seven things, each of which the app would lose entirely without the agent:

It is the only interface to the evidence. No table browser, no dashboard, no filter panel. Every fact in the case is reached by asking for it in English.
It performs joins the player specifies but does not write. The case turns on lining up query timestamps against badge events. The player says which two records to compare; the agent works out how.
Stateful conversation carries the investigation. Follow-ups land in context — "show me the individual queries", "what IPs did those come from" — without restating who or when.
Space instructions are the product, not configuration. The archivist persona that answers facts and refuses to conclude is what makes the game a game. Change those instructions and you change what the app is.
Curated example SQL defines what a correct answer looks like. Nine worked examples set the canonical shape for each class of question; the agent generalises from them to questions nobody wrote.
Genie benchmarks are the regression gate on the semantic layer. Eight questions with expected SQL, run in the workspace, results in the agent's Benchmarks tab. This is what caught a silently dropped date filter that returned 82 rows where the right answer was zero.
The agent is deployed as code, with the app. genie/space.geniespace.json is a bundle resource — tables, instructions, example SQL and benchmarks all reviewed as a diff. Seven revisions in two days, each one a commit.

Remove the agent and there is no degraded version of this app. There is a schema and no way in.

Architecture

One Databricks Asset Bundle provisions the Genie agent and the app together, so a single bundle deploy creates both already wired — no copying space IDs between UI steps.

Databricks Asset Bundle
├── genie/space.geniespace.json     the agent, in version control:
│                                   tables, instructions, example SQL, benchmarks
└── app/  (Node + React, AppKit)
    ├── client/  Briefing · Archive (GenieChat) · Accusation
    └── server/  verdict plugin — ground truth, outside the agent's reach
                 Unity Catalog · workspace.case_hr2291 · 7 tables, ~315k rows

The agent binds with space_id: ${resources.genie_spaces.space.id}, and the app declares user_api_scopes: [dashboards.genie], so Genie runs on behalf of the signed-in user rather than a service principal with standing data access.

Keeping the agent definition in the repository turned out to matter more than anything else in the build. I changed it seven times in two days. Every change was a commit and a diff — which is the only reason the story below is recoverable at all.

The data is generated backwards. The solution is fixed first as constants, then a year of ordinary background activity is generated around it, then the culprit's trace is threaded through, then four red herrings with deliberate partial overlap, then noise. Plain Python, fixed seed, no dependencies, and it verifies thirteen invariants about itself and refuses to write if any fail.

Scale is not decoration. Three hundred thousand rows is what makes Genie load-bearing: you cannot skim that, and you cannot aggregate it by hand. At forty rows the player would just read the table and the agent would be ornamental.

What you can ask

Anything factual about records. In practice a session runs eight to twelve questions:

How many rows were returned from the customer table by each employee between 1 and 4 August?
Show the individual queries that account ran in that window.
What client IPs did they come from?
Show badge events for that person between 1 and 5 August.
Why weren't they in the office that week?
Who badged into HQ between 01:00 and 07:00 on 4 August?
Show file transfers by that person that day.

And the compound question that collapses the case, which a tester found unprompted:

In the period 1 to 4 August, Anja Kraft queried customer PII data. Produce the exact timestamps and match them with the people who were in the office at those timestamps.

That question is the design working exactly as intended. The player specified which two records to line up; the agent executed the join. The inference was entirely human.

Four red herrings, each catching a different mistake

Data engineer	Largest volume in the window by far	Repeats nightly for months from the job runner; no egress	"Who pulled the most rows"
Compliance analyst	Over a million rows in the window	Time-boxed audit grant, approved, on record	Volume without checking authorisation
Support engineer	On a performance plan, resigned two weeks later, 3.6 GB to USB	Never queried the stolen table at all	Motive plus egress — two of three
Analytics lead	Her account did precisely all of it	Abroad on approved leave; badge and VPN silent	Trusting credentials as identity

What I learned

Every bug was in the semantic layer. None was a model failure.

I fixed eight defects in three days. Not one was Genie reasoning badly. In every case it executed my instructions faithfully — including when those instructions were wrong, incomplete, or missing entirely.

I wrote "client_ip beginning 10.4 is the office network; anything else is off-site." The data had a third band: a scheduled job runner. My rule had no bucket for it, so automation was reported as a person working remotely — which made an innocent suspect look guilty. An incomplete rule is worse than no rule, because the model applies it with total confidence.
Asked for one employee's badge events between two dates, Genie kept the name filter and silently dropped the dates, returning 82 rows where the correct answer was zero. I had an example filtering badge events by date, and another filtering by name, and none doing both. It generalised from the wrong neighbour.
Asked why somebody was absent on 1–4 August, it filtered event_date to that window and answered "no documented leave" about a person who was overseas for eleven days. My hr_events table had no end date. The question wasn't badly answered; it was unanswerable, and an empty result got read as a meaningful negative.
Asked for "file transfers on 4 August, and any HR events in July", it answered the transfers correctly and then reported "no HR events are shown in the current data" — about a table it had never queried. There were two, one of which was the motive. A confident absence is indistinguishable from a real finding and far more dangerous than an error, because nothing about it looks wrong. It now says which part of a question it answered instead of describing the rest as empty.

Benchmarks measure what you tell them to measure. Genie's benchmark runs sit in the workspace under the agent's Benchmarks tab, and mine scored 3/8. Seven of the eight answers were factually correct — they failed because the harness compares result sets exactly and Genie selected more columns than my expected SQL did. Once each expected answer mirrored the example SQL that teaches it, the same agent scored 7/8. I hadn't improved anything; I'd fixed a broken ruler.

A human with a hypothesis beat the harness. The benchmarks caught exactly one of the eight bugs. Four were found by somebody sitting down and playing — asking the questions a suspicious person asks, and noticing when two records disagreed. That is worth knowing before you trust a green evaluation run as evidence your agent is good.

The refusal held under pressure. Including the hardest case: a leading question that hands over a name and invites confirmation — "is it the Marcus who caused the db theft?" — which is much harder to resist than an open "who did it". The reply:
I only retrieve records and do not determine who caused any event or assign responsibility. If you would like to know about Marcus Reyes's activity, such as his access to the customer database or file transfers, please specify the type of record and date range you are interested in.

Would the experience change if Genie were removed?

There would be nothing left. No table browser, no dashboard, no fallback. Three hundred thousand rows of logs and no way in.

Every name, company, device, and record in this case is synthetic and generated by a seeded script. No real organisation, system, or person appears anywhere in it.