CosmosGenie — Your Universe, Answered (Genie-Powered App Challenge )

[sudiptob-DA]

My son asked me one evening whether any asteroids were going to hit Earth. The
data to answer him exists — NASA publishes it daily, for free — but it lives in
JSON behind API keys, in units like astronomical units and X-ray flux classes,
built for people who already know what they're looking for. A curious ten-year-old
is not that person.

CosmosGenie fixes the interface, not the data. Ask anything about space in plain
English — asteroids, eclipses, the Moon, planetary line-ups, rocket launches —
and it queries live NASA, USNO and JPL data and answers in a sentence.

**How it's built.**

Eight free public APIs feed a full medallion architecture on
Databricks Free Edition. Two ingestion paths — standalone notebooks on a Lakeflow
Job, and a Lakeflow Spark Declarative Pipeline with bronze → silver → gold and
data-quality expectations — land eight silver tables (the source of record) and
four gold tables tuned for Genie. 

[ Free Public APIs ] [ Databricks Free Edition ]

NASA NeoWs ─────┐ Lakeflow SDP Pipeline
NASA DONKI ─────┤ ┌─ bronze/ (raw API pull)
├──► SDP Pipeline ────►├─ silver/ (clean + DQ expectations)
│ └─ gold/ (business logic, KPIs)
USNO Moon ─────┐
JPL / Curated ───┤
NASA Eclipse ────┼──► Lakeflow Job ────► Delta Tables (cosmos.space.*)
The Space Devs ──┤
Spaceflight News ┘
│
┌─────────┴──────────┐
Genie Space
└─────────┬──────────┘
Streamlit App

**The app** is Streamlit on Databricks Apps with the Genie Agent attached as a
resource:

An aurora theme, a live KPI bar, an interactive 12-month events timeline
where clicking an event asks a question, and threaded chat that returns prose, the
generated SQL, a table and a tailored visual.

**Why Genie.**

Remove it and what's left is four numbers and a timeline — no
dashboard, no filter panel, no pre-built report. Every ranking and caveat is
generated live from a question nobody wrote in advance. Almost all the effort went
into the semantic layer: column comments, instructions and sample questions.

What I learned.

The natural-language part is only as good as the semantic layer behind it.The payoff is that once Genie understands the data this well, it reliably handles questions I never anticipated and never wrote an example for. It even explains its own reasoning. The lesson: the model isn't the hard part — describing your data clearly is. Invest there, and the natural-language experience takes care of itself.

[Khasim_1]

Hi @sudiptob-DA,

This is a fantastic use case for the Genie Agent. You’ve perfectly illustrated the most important lesson in modern AI-augmented data platforms: the quality of the natural language experience is entirely dependent on the quality of the semantic layer.

I really like how you’ve structured your ingestion strategy—using both Lakeflow Jobs and Spark Declarative Pipelines (SDP) to manage the complexity of eight disparate API sources. It’s a great example of using the right tool for the specific latency requirements of each data stream.

Since you’re using Genie to handle unpredictable, ad-hoc queries, how are you managing the "Golden" layer optimization to balance latency with query accuracy? I’m particularly curious if you found that specific column-level metadata (like descriptions and constraints) played a bigger role in Genie’s accuracy than the data-quality expectations themselves?