The setup every operations analyst knows
A procurement lead asks what sounds like a simple question:
"Of the orders that came in over the last hour, which ones are from customers who are close to their credit limit?"
Answering it today takes three things: someone who can query the Kafka topic the orders flow through, someone who knows where customer credit data lives in the lakehouse, and a spreadsheet to join the two halves by hand. The orders are fresh, they haven't landed in the lake yet and perhaps will not land for another hour or two. The credit limits are grounded, historical, curated and governed. Two different data realities, no single human-friendly interface across them, and a human stuck in the middle as the integration layer.
This post isn't about connecting Kafka to Databricks — that plumbing is incidental. It's about what becomes possible when a supervisor agent can reach both realities at once and answer the question itself.
This is a context engineering problem
An agent that acts on operations needs two kinds of context, and neither is sufficient alone:
- Fresh context — what is happening right now, in the stream. Lenses is a UI and SQL-powered pane of glass over a multi-vendor, multi-Kafka estate. Lenses MCP for Kafka exposes the live Kafka environment as tools: list topics, read and filter messages, introspect schemas, perform live analysis and counts.
- Grounded context — what is true and curated about the business. A Databricks Genie space exposes the lakehouse as natural-language SQL over governed Unity Catalog Delta tables.
Put the Agentbricks Supervisor Agent in front of both. It supports multi-tool reasoning and it decomposes a question, picks the right source for each part, and synthesizes the answer. Crucially, you never tell it "use Lenses for Kafka, Genie for Delta." It chooses based on the question. Add another MCP server later: Slack, a CRM, an internal API and the agent inherits the capability with no retraining.
A side benefit: explore the stream before you commit to a Spark job
There's a practical payoff beyond answering analyst questions. Today, to understand what's in a stream, its shape, its edge cases, whether the data even supports the application you want to build, teams often create a Spark job purely for exploration, with a lot of logging.
With a Lenses MCP server attached, the agent can explore the live topic conversationally: "what does this stream actually contain, and is it clean enough to build on?" and you decide what to build before you commit to building it. The intent isn't fewer Spark jobs, it's letting teams move faster on exploration so they ship the right jobs, and ultimately more of them.
The demo: Four rounds, building from grounded to fresh to enrichment and action
The pattern is most convincing as a progression. This story is about a B2B hardware distributor attempting to detect high-risk orders based on their remaining credit: live order events flow through a Kafka topic (orders-v2) surfaced by Lenses MCP, customer credit, inventory, and historical orders live in Delta tables behind a Genie space. The supervisor lives in the Databricks AI Playground with both attached.
Round 1 — Grounded only (Genie). "What's Acme Construction's credit limit and how much have they ordered year to date?" The agent calls Genie, Genie translates to SQL over Delta. Establishes: it knows the customer.
Round 2 — Fresh only (Lenses MCP). "What are their last five orders that landed in Kafka, and who placed them?" The agent calls Lenses MCP and returns live messages from orders-v2. Establishes: it knows what's happening this minute.
The order matters: here's what you already know about this customer, here's what's happening right now. That sets up the only question that needs both.
Round 3 — Multi-hop, both sources (the join). "Of the orders that came in over the last hour, which are from customers within 10% of their credit limit?" The agent calls Lenses MCP for the last hour's orders, calls Genie for credit limits and balances, and joins the two in its own reasoning. This is the part you cannot do with a single tool, no Spark job, no glue code, no copy-paste.
Round 4 — Taking action. "Draft an email to the account manager for the highest-risk orders." With an email/Slack tool attached, the agent moves from analysis to intervention — pulling the right customer, order detail, and risk reasoning into a coherent draft. This is also the perfect segue to the part most production conversations skip over: governance.
Putting this into production: governance is the real conversation
A supervisor with simultaneous access to live order events and customer credit data, plus the ability to send an email, is a textbook lethal trifecta: private data, potentially untrusted content, and external action. That combination is exactly where things go wrong: the agent could become poisoned and sensitive data could be exfiltrated. If it isn't governed, it's where this being a joint architecture matters, because each platform owns half of the safety story.
- On the Kafka half — Lenses. Role-based access control over topics and the actions the agent can take on them, PII masking on sensitive fields, and audit logging of what the agent reads. The agent only sees the streams and fields it's permitted to, and every access is recorded. Authentication and authorization are handled by OAuth, so an agent is tied to its human operator’s identity.
- On the lakehouse half, Unity Catalog provides the equivalent controls for Delta and beyond: fine-grained permissions, row and column controls, lineage, and auditability over the credit and customer data Genie exposes. It also governs the agent itself and, through Unity AI Gateway, the external MCP endpoints it can reach. That matters because the agent runs with on-behalf-of (OBO) authentication, so its access is scoped to the identity and permissions of the human operator rather than a broad shared service account. In practice, both the agent’s SQL and its tool use run inside the same policy envelope.
Together, that means the supervisor operates inside enforced boundaries across both streaming and lakehouse systems, with an audit trail spanning both. That is the difference between a compelling demo and something you would actually allow anywhere near production credit data. For teams evaluating this architecture, governance should be the first design conversation, not the plumbing.
The unlock
The pattern doesn't replace Spark, Delta, or properly architected data platforms. It changes who can ask questions of them, and how fast and it gives an agent the two kinds of context it needs to actually act: fresh from the stream, grounded from the lakehouse, governed on both sides.
Two data realities, one question, no copy-paste.
Lenses MCP server for Kafka: https://lenses.io/blog/lenses-mcp-new-era-in-ai-enablement-for-streaming-app-dev
The Challenges in Productionising MCP Servers: https://lenses.io/blog/mcp-server-production-security-challenges
