Challenge Track: Track A — Real-World Problem Solver
Built on: Databricks Free Edition
Large enterprises rarely have a reporting problem because they lack reports. More often, they have too many — but the critical problem is determining which reports are genuinely needed, which can be consolidated or retired, and which require careful review before any action is taken.
As BI estates grow across platforms such as OBIEE, Tableau, Cognos, and analytical cubes, reports accumulate over time. Some are duplicates or slight variants. Some were created for one-time needs. Others are no longer actively used.
But rationalization is not as simple as finding an unused report and deleting it.
A low-usage report may still support an executive process, regulatory requirement, customer commitment, financial dependency, or another business-critical function.
That creates the real question:
How can business and technical teams understand what should be kept, consolidated, reviewed, or retired — without repeatedly going back to spreadsheets, SQL, and manual analysis?
This real enterprise problem inspired my submission for the Databricks Genie-Powered App Challenge: BI Rationalization Genie.
For the challenge, I built a Databricks App with a Genie Agent at the center of the analytical experience.
The goal was not just to replace SQL with chat.
It was to let different users investigate the same governed BI estate naturally — starting with a high-level question, drilling into why a decision was made, and then looking for broader patterns without having to predefine every analytical path.
The application is designed for BI leaders and executives, business and report owners, and technical teams responsible for rationalization and modernization.
The Data Behind the App
For the challenge, the inventory contains 300 reports across four BI platforms: OBIEE, Tableau, Cognos, and AAS.
The curated inventory contains both report metadata and rationalization evidence, including fields such as:
- source platform
- business criticality
- decision rule
- proposed action
- final status
- mandatory category
- consolidation target
- usage and other report-level indicators
The outcome of the upstream rationalization process is one of four dispositions:
KEEP | CONSOLIDATE | RETIRE | REVIEW
A short note on the upstream process
The final inventory does require processing before Genie uses it.
At a high level, the rationalization process combines BI metadata and usage signals, evaluates rationalization indicators such as report duplication, inactivity, one-time reporting patterns, hardcoded logic, filter variants, and column variants, and then applies business-criticality safeguards.
I intentionally kept these governed business rules outside Genie.
The rationalization process creates evidence and recommendation. Genie lets users investigate that evidence conversationally.
This separation was important to me. I did not want an AI agent to invent a retirement decision that should be governed by explicit business rules.
Architecture and Data Flow
The application follows a simple flow:
The curated inventory provides the analytical foundation.
The Genie Agent contains the business context needed to interpret questions about criticality, rationalization rules, proposed actions, review status, and other inventory concepts.
The Databricks App provides the user-facing experience for asking those questions and continuing the conversation.
One Interface, Different Questions
One thing I like about this pattern is that I did not need to design a different dashboard for every audience.
The interface stays the same.
The question changes.
An executive or BI leader might ask:
Which platforms should we prioritize for rationalization?
A business or report owner might ask:
Why does this critical report require REVIEW instead of being retired?
A data or BI engineer might ask:
Which decision rule triggered the recommendation, and what is the consolidation target?
The value is not role-specific screens.
It is allowing each user to follow the analytical path that matters to them using the same governed data.
Why Genie Is at the Core
This was the most important question I asked myself while building the app:
If Genie disappeared, what would change?
The rationalization recommendations would still exist in the inventory.
But the main experience would fundamentally change.
Users would return to predefined dashboards, spreadsheet filters, SQL queries, or requests to technical teams whenever they wanted to explore the estate in a way that had not already been designed for them.
The clearest way to demonstrate the value of Genie is through the different questions users can ask from the same governed BI inventory.
Executive View — Start with the Estate
An executive or BI leader can begin with a high-level question:
Q1. What does the overall BI estate look like across KEEP, RETIRE, CONSOLIDATE, and REVIEW, broken down by source platform?
Genie summarizes the estate across platforms and rationalization outcomes, giving leadership a quick view of where reports are being retained, consolidated, retired, or held for review.
From there, the user can move from visibility to prioritization:
Q2. Which source platforms show the largest rationalization opportunity based on retirement, consolidation, and review patterns? Explain using only the available data.
Instead of requiring a predefined dashboard for every possible comparison, Genie can analyze the governed inventory across source platform, proposed action, final status, criticality, and rationalization patterns.
This gives leadership a way to explore the estate based on the question they need answered at that moment.
Technical View — Investigate the Decision
Technical users can take the conversation deeper.
I asked:
Which critical reports were recommended for retirement or consolidation but require manual review instead?
Genie identified the affected reports and explained the relationship between the original rationalization recommendation, business criticality, and the final REVIEW status.
The next turn demonstrates an important part of the conversational experience.
I did not repeat the population report or rebuild the previous filters.
I simply asked:
Of those, explain which ones were originally consolidation candidates, what decision rule triggered each recommendation, what target report they consolidate into, and why criticality changed the final status to REVIEW.
Genie understood what “those” referred to from the previous turn.
It continued from the existing conversation, narrowed the analysis to the consolidation candidates, connected each recommendation to its rationalization rule and target report, and explained why the criticality safeguard prevented automatic consolidation.
The user does not need to know fields such as critical_flag, decision_rules, target_report_id, proposed_action, or final_status.
They can continue the investigation using business language.
From Investigation to Engineering Prioritization
The analytical path can then shift again.
A technical team preparing for modernization can ask:
Based on the rationalization patterns, which platforms and decision-rule categories should the engineering team investigate first for migration, consolidation, or retirement? Use only measurable evidence from the available data and do not assume timelines, effort, or business impact.
This question moves beyond identifying individual reports.
Genie can compare measurable patterns across the estate—such as report volume, rationalization actions, decision rules, criticality, and action readiness—to help engineering teams determine where deeper investigation should begin.
Importantly, Genie is instructed not to invent implementation timelines, effort estimates, or business impact when those facts are not present in the data.
This is the experience I wanted to create.
The same governed inventory supports several levels of investigation:
What does the estate look like?
→ Where are the largest rationalization opportunities?
→ Which reports require attention and why?
→ What evidence produced those decisions?
→ Where should engineering investigate next?
The analytical path is not hard-coded into a collection of separate dashboards.
It emerges from the user's questions and can continue as the investigation evolves.
That is where Genie becomes load-bearing in this application.
What Else Can Users Ask?
The same inventory also supports questions such as:
- Which reports appear to be one-time or ad-hoc reports?
- Which reports contain hardcoded dates?
- Which reports can be consolidated, and what are their target reports?
- How do rationalization outcomes differ across BI platforms?
- Which critical reports prevent an otherwise automated rationalization action?
The user does not need to understand the schema before beginning the analysis.
They start with the business question.
Building for Accuracy Was a Bigger Part of the Work Than I Expected
Connecting Genie to the inventory was not the difficult part.
Getting consistently correct answers was where I spent more time.
I created benchmark questions around important rationalization scenarios and compared Genie's responses with the expected results.
After iterating on the semantic context and testing the generated queries, I completed a clean benchmark run in which all five benchmark questions passed.
The benchmark set covered important rationalization scenarios such as critical-report safeguards, retirement recommendations, consolidation candidates, one-time reports, and hardcoded-date patterns, giving me a repeatable way to validate Genie’s accuracy against known expected results.
One debugging habit became particularly useful:
When Genie gave me an incorrect answer, I stopped immediately adding more instructions.
Instead, I checked:
- Is the underlying data correct?
- What SQL did Genie generate?
- Is the business terminology clear?
- Does Genie have the semantic context needed to interpret the question?
- Is my expected benchmark answer itself defined correctly?
That process led to one of my most useful learnings from the challenge.
Instructions and evaluation notes are not the same thing
At first, when an evaluation failed, the natural reaction was to think: I need another instruction.
That is not always the right fix.
I started using a much simpler distinction:
Instructions help guide Genie. Evaluation notes help evaluate Genie.
If Genie repeatedly needs business context to interpret future questions correctly, that belongs in its instructions or semantic knowledge.
If the issue is defining what the benchmark judge should consider a correct response, that belongs in the evaluation note.
This small distinction helped me avoid continuously adding instructions just to make an evaluation pass.
I also learned that inspecting the generated SQL can often tell you more about an accuracy problem than adding another paragraph of prompting.
Another Lesson: Deployed Does Not Always Mean Accessible
The app integration gave me another very practical lesson.
At one point, the Databricks App was successfully deployed and running, but that did not automatically mean the complete Genie experience was accessible.
The application identity also needed the appropriate access to the Genie Agent and the required underlying resources.
That changed how I thought about testing the solution:
Test the Genie Agent. Test the app. Then test the authorization path between them.
For an enterprise implementation, the same principle extends to governance.
The conversational interface should make analytics easier to use, but it should not create a shortcut around the controls protecting the underlying data. Access to the app, Genie, and the governed inventory should continue to follow the appropriate Databricks authorization and Unity Catalog controls.
What I Took Away From the Challenge
Three lessons stand out for me.
- Do not treat every Genie accuracy problem as a prompting problem.
Look at the data, generated SQL, semantics, and expected answer first. - Conversation changes the value of analytics.
The useful part is not only asking the first natural-language question. It is being able to continue from that answer, change analytical direction, and explore something that was not predefined in the UI. - A working Genie Agent and a working Genie-powered application are two different milestones.
Integration, authorization, conversation handling, and application behavior need their own testing.
Closing Thought
I started this project with a report-rationalization problem.
The upstream process can already calculate governed recommendations such as KEEP, CONSOLIDATE, RETIRE, and REVIEW.
But a recommendation sitting in a table is not the same as making that information easy for people to investigate.
That is the role Genie plays in this application.
A BI leader can start at the estate level.
A report owner can ask why a particular decision was made.
A technical user can drill into the rule and supporting evidence.
And they can move between those questions through a conversation rather than waiting for another query, spreadsheet, or dashboard to be built.
The rationalization process creates evidence. Genie turns that evidence into an investigation.
#Databricks #DatabricksGenie #DatabricksApps #DataAI #BusinessIntelligence #DataModernization #GeniePoweredAppChallenge
