[CUSTOMER BLOG] How S&P Global Energy Made Its Structured Data Estate Conversational with Databricks

[debaprasad]

If you have ever tried to make a large, complex structured data estate available to AI agents and assistants, you have likely run into the same wall we did: agents are only as good as the context they can reach, and enterprise data rarely lives in one neat, well-documented place.

At S&P Global Energy, our data spans Chemicals, Crude Oil, Refined Products, Gas & Power, Liquified Natural Gas (LNG) and more — and each commodity is itself a rich family of datasets. LNG alone includes facility specifications, cargos , outages, supply and demand fundamentals, netbacks, historical and forecast prices, and contracts. Chemicals spans capacity, production, utilization, trade, demand by end use and by derivative, inventory change, and country- and region-level supply–demand balances. Our other commodities follow similar patterns. This data lives across Databricks and several non-Databricks sources.

Our goal was ambitious but simple to state: make our entire structured data estate available for external consumption by AI agents through the Model Context Protocol (MCP) — so that our customers’ agents and assistants, as well as our own, could ask questions in natural language and get trusted, governed answers.

We evaluated several approaches. What worked best for us, by a wide margin, was Databricks Genie Agents, exposed as managed MCP servers, composed into domain-specific bundles with an MCP proxy layer.

In this post, you’ll learn:

• How our subject matter experts (SMEs) curate focused Genie Agents — one per dataset group within each commodity — without writing a single line of agent code.

• How each Genie Agent automatically becomes a governed MCP server, ready to plug into any MCP-compatible client or agent.

• How we use a FastMCP-based proxy to compose multiple Genie MCP servers into composite endpoints for cross-domain questions.

• Why this architecture dramatically shortened our time to market for AI-powered data products.

‘Our collaboration with Databricks is accelerating our AI strategy. By leveraging tools like Genie Agents, we are fundamentally changing how our Energy experts interact with data, enabling them to build and deploy solutions at a pace that was previously unimaginable, all within a secure environment.’ — Priyanka John, Vice President, S&P Global Energy

The Challenge: Structured Data Is Easy to Store, Hard to Converse With

Large language models are remarkably good at conversation and reasoning, but they cannot answer questions about your data unless you build a bridge to it. For structured enterprise data, that bridge has historically meant one of the following:

1. Hand-built text-to-SQL pipelines — powerful, but brittle. Every schema change, every ambiguous column name, every domain-specific metric definition (“What counts as an outage day?”) becomes an engineering task.
2. Custom APIs per use case — each new question pattern needs a new endpoint, a new sprint, a new release.
3. Exporting data into external AI tools — which duplicates data, breaks freshness, and steps outside your governance perimeter.

Each of these approaches shares the same problem: the people who understand the data best — our SMEs and analysts — are not the people building the access layer. We needed an approach where domain experts could curate and publish conversational access to data directly, engineering could standardize how agents connect, and governance stayed centralized. That is exactly what Genie Agents plus MCP gave us.

The Architecture: Genie Agents as the Semantic Layer, MCP as the Contract

Our architecture has three layers, and each layer is owned by the people best suited to it. The diagram on the following page shows the end-to-end flow — including what sits inside the S&P Global Energy network and what sits in the external client environment.

Layer 1: SMEs Curate One Genie Agent per Dataset Group

This is where the magic starts, and notably, it requires no code.

Our SMEs begin by selecting the tables relevant to a business domain:

• If the tables already live in Databricks, they use them directly through Unity Catalog.

• If the data lives in a non-Databricks source, they bring it in through Lakehouse Federation connectors — no data movement, no duplicate pipelines. The federated tables appear alongside native tables and inherit the same governance.

They then group related tables and create one Genie Agent per dataset group — not one giant agent per commodity. Each sub-category of a commodity becomes its own focused Genie. In the LNG market, as an illustrative case:

• LNG Assets & Contracts Genie — assets, operators, capacity forecasts, and long-term contracts

• LNG Cargo Genie — cargo tracking, fixtures, origins/destinations, and commercial terms

• LNG Tenders Genie — tenders with issuers, volumes, and delivery windows

• LNG Outages Genie — outages and maintenance events with capacity impact

• LNG Supply & Demand Genie — fundamentals with regional splits and scenario history

• LNG Netbacks Genie — netbacks from hub prices, freight, boil-off, and losses

• LNG Prices Genie — historical and forecast price curves

Every other commodity follows the same pattern with its own sub-categories. Chemicals, for instance, has group-level Genies for capacity, production, capacity utilization, trade, demand by end use and by derivative, inventory change, and country- and region-level supply–demand balances; Crude Oil, Refined Products, and Gas & Power are organized similarly. The result is a fleet of small, sharply scoped Genie Agents rather than a handful of sprawling disconnected AI tools.

Inside each agent, SMEs add the context that makes text-to-SQL actually work in the real world: descriptions of tables and columns, example queries, trusted assets for high-stakes metrics, and business definitions (for example “floating storage is defined as cargoes idling for 3 days or more in vessels travelling below a threshold speed”). This is the step that generic text-to-SQL solutions skip — and it is the step that determines whether users trust the answers.

The key organizational insight: curation became a domain activity, not an engineering activity. The person who knows what “floating storage” means in an LNG context is the person teaching a Genie what it means.

Layer 2: Every Genie Agent Is Automatically an MCP Server

Here is where Databricks did the heavy lifting for us. Each Genie Agent is exposed as a Databricks managed MCP server out of the box, at an endpoint of the form:

https://<workspace-hostname>/api/2.0/mcp/genie/{genie_agent_id}

There is nothing to deploy and nothing to host. Each server exposes a small, clean tool surface — essentially two tools per agent:

1. A query tool (genie_query_space) — the agent submits a natural language question to the agent.
2. A response tool (genie_poll_response) — the agent polls with the same conversation and message ID to retrieve the full response once it's ready, including the generated SQL and result set.

This two-tool, ask-then-poll pattern turns out to be a great fit for agentic workloads: questions run asynchronously against a SQL warehouse, and the agent polls with the conversation and message ID returned by the query tool until the response is ready..

Just as importantly, these managed servers are governed by Unity Catalog. An agent — or the user behind it — can only reach the agents and underlying tables they have permission to see. Authentication is handled by the platform. We did not have to build a security layer around our AI access; we inherited the one we already had.

Layer 3: Composing Group Genies into Commodity Bundles with a FastMCP Proxy

One Genie Agent per dataset group keeps each agent focused and accurate. But real business questions routinely cross groups: “How did the recent outages at Sabine Pass affect cargo premiums into Asia?” touches the Outages and Cargo Genies at once, and cross-commodity questions like “How are naphtha prices affecting chemicals production margins?” reach across Refined Products and Chemicals.

Rather than building one giant agent (which degrades answer quality) or forcing every client to configure a dozen separate servers, we used FastMCP’s proxy and composition capabilities to create composite MCP endpoints — typically one per commodity, mounting that commodity’s group-level Genie MCP servers behind a single server with name-spaced tools. Higher-level composites can bundle several commodities the same way:

from fastmcp import FastMCP 

# Each group-level Genie Agent is a managed MCP server on Databricks 

cargo = FastMCP.as_proxy(genie_mcp_config(“lng_cargo_agent_id”), name=“cargo”) 
outages = FastMCP.as_proxy(genie_mcp_config(“lng_outages_agent_id”), name=“outages”)
netbacks = FastMCP.as_proxy(genie_mcp_config(“lng_netbacks_agent_id”), name=“netbacks”) 


# Compose the group Genies into one commodity bundle 

lng = FastMCP(name=“lng-composite”) 
lng.mount(cargo, prefix=“cargo”) 
lng.mount(outages, prefix=“outages”) 
lng.mount(netbacks, prefix=“netbacks”) 

# The same pattern repeats for Chemicals, Crude Oil, Refined Products, Coal …

(Illustrative snippet — adapt to your FastMCP version and auth setup.)

The result: an agent connects to one composite endpoint per commodity and sees a curated set of group tools — cargo_genie_query_agent, outages_genie_query_agent, netbacks_genie_query_agent, and so on, each paired with its genie_poll_response counterpart. The agent’s LLM decides which group Genie to route a question to, or fans a cross-group question out across several, then synthesizes the results.

This gave us the best of both worlds: narrow, high-accuracy group-level Genie Agents underneath, and broad, commodity- and estate-wide conversational access on top.

What Changed for the Business

For our business stakeholders, the technical details above translate into a few very tangible outcomes.

Time to market collapsed. Previously, standing up a new conversational data experience meant a full development cycle: requirements, API design, text-to-SQL engineering, testing, deployment. With this architecture, launching a new dataset group — or an entire commodity — means an SME creates and curates the corresponding Genie Agents — the MCP endpoint exists the moment the agent does.

SMEs became publishers, not requesters. The domain experts who understand LNG cargoes or chemicals supply–demand balances no longer file tickets to get their data exposed; they curate a Genie Agent and it is live. Engineering effort shifted from building bespoke access layers to maintaining one thin, reusable proxy layer.

Governance came for free. Every question an agent asks runs through Unity Catalog permissions, on governed tables (native or federated), with full auditability. Making data available to AI did not mean making it available outside our controls.

One integration pattern, many consumers — inside and outside the company. Because MCP is an open standard, the same composite endpoints serve our internal agents, our customer-facing AI experiences, and — critically — our external customers, who can connect their own MCP-compatible agents and assistants directly to governed S&P Global Energy data. We built the bridge once; every MCP client, internal or external, can cross it.

Lessons Learned and Best Practices

A few practical takeaways from our journey, for teams considering a similar path:

1. Keep Genie Agents narrow and well-curated. Answer quality is highest when an agent covers one specific data domain with clear instructions and example queries. Resist the temptation to build one agent per commodity — or worse, one agent to rule them all. Bring together multiple data domains at the MCP layer instead.
2. Use Lakehouse Federation before you build pipelines. For non-Databricks sources, federation got us to “conversational” without a single new ETL job. You can always materialize hot paths later.
3. Invest in the semantic layer. Column descriptions, business definitions, and trusted example queries are what separate a demo from a product. This is SME time well spent.
4. Namespace your composite tools clearly. When an agent sees tools from many group Genies, prefixes like cargo_ and outages_ help the LLM route questions correctly.
5. Measure trust. We tracked how often SMEs agreed with Genie’s generated SQL during curation — it is the best leading indicator of whether business users will adopt the experience.

Conclusion

Our goal was to make our entire structured data estate — spanning LNG, Chemicals, Crude Oil, Refined Products, Gas & Power, and more — available to AI agents: securely, accurately, and fast. With Databricks Genie Agents as the SME-curated semantic layer, managed MCP servers as the zero-deployment integration contract, and a FastMCP proxy for cross-domain composition, we achieved exactly that:

• Speed: New conversational data domains go live in days, not development cycles — supercharging our time to market.

• Accuracy: Domain-scoped, SME-curated agents deliver answers business users actually trust.

• Governance: Unity Catalog secures every question, on native and federated data alike, with no parallel security stack to maintain.

• Openness: One MCP-standard bridge serves every current and future agent, internal or external.

The deeper shift, though, is organizational: the people who understand the data are now the people who publish access to it. That, more than any single technology, is what turned our structured data from something users query into something they can simply talk to.