Genie Metadata QA: Hands-On Testing Tips

[Asviri]

Introduction

If Genie misunderstands your data, your users will lose trust fast, so spot issues early by stress-testing Genie with smart prompts before rollout. Think of this as QA for your metadata: quick, focused, and business-critical. Genie is Databricks’ conversational experience, powered by generative AI, for business teams to go beyond BI dashboards and self-serve insights in real time from their data using natural language.

As creators/admins of the Genie space, we usually worry about several things. These can be summarised into several main blocks: 

• Accuracy and comprehension
• Performance and stability
• Cost and resource management
• Rollout and adoption

This article focuses on the first block, improving answer accuracy and comprehension through prompt-based testing and metadata diagnosis techniques.

Standard Launch Process And Potential Issues

Genie spaces are often built by Data Engineers, Analysts, or Business Teams, each with a different understanding of the best practices and how data should be used. This can cause subtle misalignments, such as using “customers” and “clients” interchangeably or calculating delivery times from the order date instead of the shipment date. To prevent these, always include a domain expert review followed by quick user testing before rollout. At this stage, Genie already transforms natural-language questions into SQL through metadata enrichment, meaning that even small metadata gaps can surface as incorrect results.

Rather than waiting for users to stumble on these issues, test proactively. Treat Genie’s setup as metadata QA: prompt it like a user would, stress its understanding, and look for where it falters. The following sections outline how to design those diagnostic questions to uncover weak points early and tune your metadata for accuracy.

Metadata Testing Framework 

Let’s walk through how we can begin our testing. The table below serves as quick navigation in case you want to skip ahead, but we recommend going through the rest of the content - as with all non-deterministic systems, root cause analysis can be tricky and requires several iterations.

Core Debugging Criteria	Symptoms in Usage
Understanding Wrong or Inconsistent Results	Genie produces incorrect or inconsistent numbers, where data appears to come from the wrong table, joins are mismatched, or metrics don’t align with known business totals.
Diagnosing Missing or Hallucinated Data	Genie references fields, tables, or filter values that don’t exist in your catalog, or fails to use available ones. Responses may include outdated, invented, or missing data points.
Testing for Ambiguity and Clarity	Genie confuses similar terms (e.g., client vs customer), misinterprets synonyms, or gives vague, generic summaries instead of specific business results.
Checking Coverage and Scope	Genie can’t calculate certain metrics, only returns partial results, or reports missing fields because data, joins, or permissions are incomplete.

Testing Genie’s Understanding

Before jumping into specific problems, test Genie the way users will. Ask natural questions that mimic real workflows. The goal is not to “trick” Genie, but to surface gaps in how it interprets your metadata.

Quick guidelines:

• Focus on frequent, high-impact business questions.
• Phrase queries naturally, in the language your users would use.
• Mix short factual asks with scenario-based questions.
• Rephrase key questions 2–3 ways to test synonym and context handling.
• Observe not just the answer, but how Genie structures it, as that reveals metadata depth and logic alignment.

These prompts give an early signal of Genie’s comprehension: which tables and fields it recognises, how well it maps business terms, and where it defaults to generic logic. They’re not meant to find every error, but to guide where to focus deeper metadata tuning.

Framing the Questions

Understanding Wrong or Inconsistent Results

Ensure that the “numbers look right” to your end users, so diagnose accuracy issues.

Examples:

1. What tables are there, and how are they connected?
2. Explain the <name> dataset. What key metrics or columns can you report on?
3. Show me open pipeline deals.
4. List all pending invoices.
5. If I ask for 'GMV' or 'revenue', will you treat them as the same field?
6. Show net revenue by quarter

Hints:

• (1, 2) Verify Genie’s summary. If tables look unrelated or missing, check table descriptions and PK/FK (or join) relationships.
• (3, 4) If results point to wrong tables, align business terms (“pipeline”, “invoice”) with correct table names or use aliases.
• (5) If synonyms or abbreviations aren’t mapped (e.g., GMV ≠ revenue), add them to metadata descriptions.
• (6) If revenue roll-ups look wrong, compare SQL logic to your reference calculation and adjust sample queries or create metric views.
• Always check date and region formats; misformatted fields can cause silent aggregation errors.

Diagnosing Missing or Hallucinated Data

Ensure that Genie only uses real values and tables from your catalog. If you see fields, filters, or SQL outputs that don’t exist, test for sampling or metadata gaps.

Examples:

1. Which product statuses do you recognise? 
2. Show me all turbines where the state is exactly 'Denver'— exclude results for similar-sounding or partial matches. And check the results if I allow case-insensitive search
3. Retrieve all customers from ‘vip_clients’ and show their subscription plan. (N.b. subscription_plan doesn’t exist or is related to another table, so you’re testing for hallucination)
4. What metrics can’t you calculate? (N.b. the dataset has the metric logic/column)

Hints:

• (1, 2) If Genie suggests case-insensitive values and you expect case-sensitive ones, enable value indexing or refresh sampling stats for those columns. Or provide straightforward instructions on how to handle case-sensitivity
• (3) If Genie references dropped or renamed assets, re-sync metadata and check Unity Catalog lineage for duplicates.
• (4) If Genie says it can’t calculate a metric, confirm whether the underlying column or derived logic exists.
• Refresh prompt matching regularly and compare Genie’s visible fields to catalog definitions — hallucinations often indicate stale metadata.

Testing for Ambiguity and Clarity

Ensure that Genie correctly interprets your questions - not just the words, but their intent. Ambiguity in terms, synonyms, or phrasing often leads to vague, incomplete, or misleading answers.

Examples:

1. Compare client and customer data.  (N.B. It can be  device_status and sensor_status for an IoT use case) Does the output return values from different or the same tables?
2. List device readings with their location.
3. Show employee performance stats.
4. Summarise sales results, including today’s revenue. (N.b. Knowing the latest revenue update was yesterday)
5. What would you need to clarify to answer this question better?

Hints:

• ​​(1) If Genie treats “client” and “customer” as identical, clarify column names or add synonyms in metadata.
• (2) If Genie picks wrong context fields (e.g., ‘region’ instead of ‘location_xy’), tighten table descriptions or define joins.
• (3, 4) If answers are too generic, improve table descriptions and trusted SQL examples to anchor business logic.
• (5) If Genie never asks for clarification, add explicit instruction text: “Ask follow-up questions when terms overlap.”
• Review whether your column and metric names read naturally - business phrasing improves model understanding.

Checking Coverage and Scope

Ensure that Genie has access to all the data and metrics your users expect. When Genie says it can’t calculate a measure or gives partial results, the issue often lies in incomplete datasets, missing joins, or permission gaps.

Examples:

1. What metrics can’t you calculate with the current data?
2. Are there any tables or joins missing for analysing profitability?
3. Can you compute churn rate? If not, what data is missing or I don’t have access to?

Hints:

• (1) Compare Genie’s reported limits to your KPIs, as missing metrics often indicate incomplete joins or data scope.
• (2) If tables are missing, check whether they’re registered and visible in the Unity Catalog.
• (3) If Genie identifies missing fields, extend your dataset with derived columns, metric views, or calculated measures, or ensure you have access to the relevant data.
• Always check permissions - restricted columns may appear “missing.”
• Before rollout, compare Genie’s table list to your production model to confirm full data coverage.

How to Interpret Genie’s Answers - final thoughts

Treat every Genie response as feedback. Each mismatch or unexpected result signals where your metadata, joins, or instructions need refinement.

How to act on Genie’s feedback:

• Accurate outputs → Good metadata alignment. Log these queries as benchmarks. After significant data or schema changes, use these benchmarks to pinpoint accuracy drifts.
• Generic or invented values → Refresh prompt matching, update column descriptions, and verify sampling.
• Wrong relationships or missing fields → Revisit PK/FK definitions, joins, and table visibility in Unity Catalog.
• Business logic errors → Add or update example SQL, trusted assets, or metric views to anchor correct calculations.
• Persistent confusion → Rename overlapping columns or strengthen instruction text to clarify context.
• Drift over time → Capture Genie’s responses as a baseline QA log through benchmarks. Improvements over time confirm metadata and instruction quality.

Once the space performs consistently, save your best diagnostic questions as a benchmark set and re-run them before every major schema or instruction update.

This becomes your automated QA loop — a lightweight, repeatable check that keeps Genie reliable at scale.

Don’t wait for users to find what Genie doesn’t know. Probe it, tune it, and retest until the answers sound like your business. Every iteration builds trust and turns metadata QA into one of your most powerful quality gates before production.