The Trillion $ shift Supply Chains aren't ready for
Gartner predicts that by 2028, 90% of B2B buying will be AI agent intermediated, pushing over $15 trillion of B2B spend through AI agent exchanges. This is a world where AI agents negotiate, purchase, and transact on behalf of businesses. When an AI agent requests a product and expects a price, delivery date, and fulfillment plan in seconds, the supply chain behind that Supplier becomes the competitive differentiator. And autonomous supply chains that can execute at this speed and scale will be the winners.
The problem: Most existing retail supply chain systems are based on pre-defined rules that work for a pre-defined set of situations, with a vast number of anomalies routed to human intervention, which translates to response times in hours or even days. This is not an acceptable SLA in an Agentic Commerce world as the AI agent would just move on to the competitor resulting in a lost opportunity for the retailer.
The opportunity: According to a survey conducted by Accenture, autonomous supply chains have shown to deliver 27% improvement in order lead times and a 62% reduction in response times, resulting in over 5% improvement in EBITDA. What is required is a system that can react to a demand signal, sense an anomaly, reason through the alternatives to determine the best response, and execute the action on behalf of the Retail Supply Chain, all in a matter of seconds. Achieving this requires a fundamentally different architecture, one that is always-on, context-aware, self-evaluating and governed.
In this post, we describe how to build an Autonomous Supply Chain Control Engine (SCCE) on Databricks. This is an Agentic system that reasons over fulfillment alternatives, triggers execution, and continuously improves, while keeping humans in control of the decisions that matter most.
The Retail Scenario: Unplanned demand spike
A weather event along the Eastern seaboard drives a bulk order for wooden boards from a Shopping Agent acting on behalf of a contractor to a Specialty Retailer. The order volume is such that no single fulfillment center can cover it, making it necessary to make cost and delivery trade-offs to make the optimal fulfillment decision. In a standard supply chain system, this becomes an exception in a queue. The planner picks it up, pulls the logistics team together, and works through the alternatives (e.g. split the shipment, transfer inventory between FCs, expedited shipping and so on) trading unit cost against delivery confidence and downstream network impact. All this could take several hours, by which point the commitment is missed and the shopping agent has moved to a competitor.
The OODA Loop: A mental model for autonomy
But before we get into the Autonomous SCCE architecture, it is worth being precise about what "autonomous" actually means, and for that, we borrow a mental model from an unlikely place: aerial combat. Col. John Boyd, a US Air Force fighter pilot and military strategist, observed that the pilot who cycles through the OODA loop fastest wins the dogfight. OODA stands for Observe, Orient, Decide, Act: a continuous cycle of sensing the environment, contextualizing what you sense, selecting a course of action, and executing it. Victory belongs not to the pilot with the better plane, but to the one who closes the loop faster than the adversary can react.
Notice that the supply chain planner in our scenario is already running an OODA loop. She observes the exception in her queue, orients using dashboards and spreadsheets, decides in a meeting with logistics, and acts by keying the resolution into the Order Management System (OMS). The loop works. It just takes several hours, and in agentic commerce, the "adversary", the AI shopping agent with a purchase mandate, closes its loop in seconds.
This gives us a working definition: an autonomous agent is a system that observes, orients, decides, and acts in a tight, durable loop, without waiting for human prompts. Two words in that definition carry all the weight. Tight means the loop closes in seconds, not days. Durable means the loop survives failures, restarts, and long waits. For instance, it can 'sleep' awaiting a transport carrier update and resume exactly where it left off. Each phase of the loop is trivial to build, but the real value is in keeping the loop alive: durable, observable, governable is the hard engineering problem, and it is what the rest of this post is about.
The Architecture: SCCE as an OODA loop
The SCCE implements each phase of the loop with Databricks primitives. A note before we begin: this does not replace your OMS or ERP. The SCCE sits alongside your transaction systems as an intelligence layer, consuming change data capture (CDC) events and writing decisions back via API. The ERP/WMS remains the system of record.
Observe: Signal-driven activation
The Observe phase is designed to minimize the gap between an event and initiating the system response. Events like orders, inventory updates, and carrier status changes flow continuously from source systems (e.g. ERP/WMS) into Delta tables via Lakeflow Connect (CDC) or Delta Sharing. Structured Streaming monitors these events using either rule-driven anomaly thresholds or Anomaly Detection ML models deployed on Model Serving and trained via Model Training. A demand spike, inventory level approaching safety stock, a carrier delay breaching its SLA are examples of events and when the pipeline detects one, it publishes an activation event. Critically, this trigger is actor-agnostic: the same event pattern activates the agent whether the anomaly originated from a human order, an IoT sensor, or another agent.
Orient: Durable queue and memory
Activation does not mean the agent starts reasoning from a blank slate. The anomaly lands in a durable queue in Lakebase, alongside the agent's memory of prior steps. This means that if the process is interrupted, or deliberately sleeps while awaiting an external update, it resumes from where it left off rather than re-executing. The context is loaded from persistent memory in Lakebase, and the Agent orients to the latest trigger by calling tools in parallel to assemble context:
- SQL Functions governed by Unity Catalog. For instance, the Promotional Calendar function queries active and upcoming promotions to return the active set of applicable pricing and product promotions.
- Genie: Operational data is extracted from structured sources using natural language. A Network Visibility Genie space provides FC-to-ship-to mapping with transit time and cost-per-unit by lane; an Inventory Status space provides inventory positions by FC, refreshed continuously from WMS CDC feeds. A Customer Intelligence Genie space uses Customer Lifetime Value (CLV) and Churn Risk scores to determine a customer risk profile.
- Knowledge Assistant (RAG agent): Unstructured knowledge like safety stock policies by FC and SKU category, split shipment rules, expedited shipping thresholds, compliance rules like Hazmat handling and cold chain requirements and so on are retrieved from the corpus of documents.
- External Signals (via MCP Server): Real-time transit conditions like weather disruptions, port congestion, and carrier capacity constraints from external data providers are retrieved via MCP connections.
Decide: Evaluate and escalate
Using this context, the agent generates fulfillment alternatives and scores each on cost-to-serve, OTIF confidence, and second-order network impact. The ranking is not a fixed formula: for a high-LTV customer with elevated churn risk the agent will accept a higher unit cost to protect delivery confidence, whereas for a cost-sensitive segment the cheaper, slower option wins. Just as important as deciding is knowing when not to: orders above a configurable financial threshold, decisions where the top-ranked alternative's OTIF confidence falls below a floor, or any action that would breach safety stock, split-shipment, or compliance policies are escalated to the human planner via a Databricks App rather than executed.
Act: Background execution
Within guardrails, the agent triggers the fulfillment action via API to the transaction system (e.g. WMS/TMS). This decision becomes a durable memory recorded in the Lakebase table. As an off-line process, the Evaluation harness grades the decision using AI judges. And equally importantly, the impact of the decision, once it is available, is recorded. For instance, the OTIF delivery metric computed based on the actual delivery date, customer sentiment etc. are used as signals to evaluate the agent’s performance. Over time, these signals can be used as the reinforcement mechanism to improve the agent’s effectiveness.
The execution substrate
The mechanics of how that run is executed are what separate a demo from a production system, and they are worth unpacking in detail. It is evident that a fulfillment decision is not a chat completion. The decision lifecycle could extend from milliseconds, to several minutes or hours. Five design choices make that survivable.
1. Background execution mode
Agent runs are invoked in background mode: the caller submits a request and immediately receives an acknowledgement with a task identifier, rather than holding an open connection until the agent finishes. Orchestration happens server-side; the client polls for status against the task ID, or subscribes for a completion notification.
This matters for three reasons. First, synchronous inference endpoints sit behind gateway timeouts measured in minutes and any run that waits on a carrier quote or a human approval will be severed mid-flight. Second, a long-lived connection means compute pinned to an idle request; at supply chain event volumes that is a large bill for waiting. Third, and most importantly, a client disconnect stops being an outage: the run is owned by the server, not by the socket. The caller can crash, redeploy, or go home for the night, and the task continues. On Databricks, the Supervisor API pattern provides this managed orchestration with background execution, so the agent framework, not your application code, owns the run lifecycle.
2. Lakebase as the durable request queue
Background execution is only as good as the state behind it, and that state lives in Lakebase, a Postgres-compatible, fully ACID transactional store that sits inside the lakehouse. Two tables carry the system.
The task queue holds one row per anomaly: the triggering event, a status (queued, claimed, running, waiting, escalated, done, failed), the owning worker, a lease expiry, an attempt count, and a visible_after timestamp. The step log holds the checkpoints — each tool call, its result, and the agent's intermediate reasoning state — written transactionally as the run progresses.
Persisting progress rather than just outcome is the point. Recovery resumes from the last committed step instead of replaying the run from the top, which matters both for cost (no re-paying for four tool calls to get back where you were) and for correctness (no duplicate calls to an external pricing API). Every side-effecting action carries an idempotency key derived from the task ID and step index, so a retry after a partial failure cannot double-submit a fulfillment order. And durable sleep becomes trivial: for instance, an agent waiting six hours for a carrier confirmation writes visible_after = now() + 6h and releases its lease. It holds nothing while it waits.
Because Lakebase is part of the lakehouse rather than a bolted-on operational database, the same rows the runtime writes are queryable from the analytics side without an ETL hop. The scorecard, the audit query, and the agent's own working state read from one copy of the truth.
3. The always-on worker
The component that makes the system always-on rather than merely event-triggered is a long-running worker, a process hosted as a Databricks App or a continuous job — that does exactly three things in a loop: poll, claim, dispatch.
It polls the Lakebase queue for rows that are due (status = 'queued' AND visible_after <= now()) and claims them atomically. The Postgres primitive for this is SELECT ... FOR UPDATE SKIP LOCKED, which lets several workers drain the same queue concurrently without ever handing the same task to two of them and without blocking each other. On claim, the worker stamps its identity and a lease expiry, initiates the agent run in background mode, and immediately moves on — it dispatches runs, it does not host them. If a worker dies mid-flight, its leases expire and the tasks become visible to its peers, which is the difference between a crashed pod and a lost order.
This is also where the cheap gatekeeper lives. Not every event deserves an LLM. The worker applies inexpensive checks — is this signal actually off-track, has a similar task already dispatched, is now the right moment to intervene given the last outcome — and only opens the gate to the expensive reasoning loop when it should. Always-on must not mean always-thinking; running a reasoner against every event is both ruinously expensive and slower to matter. The outer loop shapes when the inner loop runs, and outcome measurements feed back to tune its thresholds over time.
Workers are stateless and horizontally scalable, because all state is in Lakebase. Scaling throughput is adding workers; there is no leader election, no in-memory queue to lose, no sticky routing.
4. Observability and auditability
An agent that moves millions of dollars in inventory has two audiences that both need to see inside it, and they need different things.
Engineers need to debug. MLflow Tracing captures the execution record of every run: a span per tool call with inputs, outputs, latency, token usage, and errors, plus the retry and resume boundaries. A failed run is inspectable after the fact rather than reproducible only in theory — you can see which tool returned malformed data at 2am, and replay from that step. Notably, this is an engineering record of what the agent did, not an exposure of hidden model chain-of-thought.
Auditors and SMEs need to justify. The Lakebase decision log carries the business record: the anomaly that arrived, the alternatives generated, the rationale for the ranking, the action taken, and the guardrail evaluation that permitted it. Every trace is keyed to its task ID, so a question that starts as a business one — why did we split this shipment? — resolves down to the exact tool calls and data reads that produced the answer.
Underneath both, Unity Catalog governs and records access: the agent's service principal reads only what it needs, PII is masked via dynamic views so the agent reasons on segments and scores rather than raw customer data, and lineage tracks which tables and functions influenced which decision. Logs are retained per organizational policy — commonly seven years for SOX-relevant decisions. MLflow Evaluation closes the loop by scoring decisions against post-delivery outcomes, which is what feeds the agent scorecard: OTIF, cost-to-serve, customer satisfaction, and network balance, measured against targets set in the S&OP process. The planner reviews that scorecard and tunes guardrails, policies, and tools accordingly — the agent improves not through fine-tuning, but through better tools, better policies, and better guardrails informed by measured outcomes.
5. Interoperating with other agents (A2A)
The demand signal in our scenario originated from another company's agent, and increasingly it will arrive as a direct agent-to-agent request rather than as an order landing in the OMS. A2A (Agent2Agent) is the emerging open protocol for exactly this: agents advertise their capabilities, exchange tasks, and track them through a lifecycle across organizational boundaries. It is complementary to MCP: MCP is how an agent calls a tool, A2A is how an agent talks to a peer.
The useful observation is that A2A's task model is the same shape as the execution substrate above: a task is submitted, acknowledged with an ID, progresses through states, and may require input before it completes. An SCCE built around a Lakebase-backed queue and background execution can therefore expose an A2A endpoint over the machinery it already has — an inbound agent request becomes another row in the same queue, subject to the same guardrails, the same escalation thresholds, and the same audit trail as an internally detected anomaly.
Anatomy of the Agent Runtime
Stepping back from the supply chain specifics, the agent at the center of this loop follows a general architecture pattern: a reasoning loop powered by models, memory, tools and resources, composed of three layers, each mapped to a Databricks primitive:
- Layer 1 — Cognition (how it thinks): Foundation models accessed as FM APIs via the AI Gateway; orchestration via the Supervisor pattern for managed, multi-agent delegation; runtime engineering via supported agent frameworks, including context engineering for managing what the model sees at each step.
- Layer 2 — Memory & State (what it remembers): Agent memory in Lakebase; workflow and agent state managed by the frameworks with Lakebase as the durable store.
- Layer 3 — Action (what it does): Tools registered as MCPs and functions in Unity Catalog; reusable, versioned Skills in Unity Catalog that capture how a task should be performed.
Cutting across all three layers is trust and governance — every block above is observed, evaluated, and guarded. Guardrails, cost controls and rate limits are enforced at the AI Gateway. MLflow provides tracing and evaluation. Unity Catalog governs every asset the agent touches. The agent cannot create or modify its own policies — only human SMEs can, via the Databricks App — and a kill switch can revert the entire system to recommend-only mode at any time.
Getting Started: The Crawl-Walk-Run Adoption Path
Deploying fully autonomous agents requires process and people transformation alongside the technology, and is best implemented in phases:
Phase 1 - Crawl: Decision Support. Start small: one product category, one region, agent in recommend-only mode. Events stream from OMS/WMS via Zerobus Ingest, Structured Streaming detects anomalies, and the agent generates ranked alternatives — but does not execute. Every recommendation surfaces to the planner via dashboard, and the metric is simple: how often does the agent's top recommendation match what the planner would have chosen? When the agent matches on 80%+ of decisions over 8 weeks, you've earned the right to Phase 2. Prerequisites: streaming pipeline from OMS/WMS, inventory data in Unity Catalog, basic CLV model deployed.
Phase 2 - Walk: Bounded Autonomy. Give the agent a lane: for orders below a configurable value threshold in the pilot region, the agent executes autonomously, with MLflow Tracing and the Lakebase decision log providing the engineering and business records described above. Everything above the threshold, below the confidence floor, or in breach of policy escalates to a human. Expand to two or three categories. The weekly scorecard review becomes the heartbeat of the system — and the number that matters most is how many hours per week your planners are getting back. Prerequisites: governance framework live, audit logging validated, human escalation workflow tested.
Phase 3 - Run: Scaled Autonomy. Expand autonomous decision-making across regions and categories. Introduce new task types aligned with the supply planner role by capturing processes as agent skills and adding tools. Human planners shift from number crunchers to agent supervisors: managing escalations, reviewing scorecards, tuning parameters, and handling true edge cases. Prerequisites: measured agent performance, organizational alignment, change management complete.
The loop is the differentiator
This blog covered a design pattern for an Always-on, Autonomous Agent with a Retail Supply Chain use-case. Nothing about this pattern is specific to Retail Fulfillment. A financial institution can run the same loop over transaction streams to stop a suspicious payment and clarify with the customer, while auto-clearing the false positives, acting before the charge settles, not just flagging it after. A manufacturer can run it over sensor telemetry so that when a machine drifts out of tolerance, the agent books the technician and orders the part on its own, escalating only when the fix would idle a production line. A healthcare payer can run it over inbound claims to pay the clean ones and hold the ambiguous ones with the missing evidence, clearing the backlog rather than re-queuing it. In each case the substrate is identical: a durable queue, an always-on worker, a reasoning agent, background execution, an audit trail - only the prompts, the tools, and the data change. As more of the economy shifts to agents transacting with agents, the advantage accrues to organizations whose systems can close the loop at machine speed without giving up control. Building these agents to be durable, observable, and governable is no longer an experiment at the edge of the org; it is fast becoming the price of staying in the game.
Why Databricks
Most autonomous, stateful Agentic Systems require stitching together separate platforms for streaming, analytics, ML, governance, and application serving. This creates integration tax, data movement latency, and governance gaps between systems. For an always-on agent, every gap is a place where the loop can silently die. The queue lives in one vendor's database, the traces in another's observability tool, the outcome metrics in the warehouse, and reconciling them becomes a standing engineering cost.
Databricks is the unified platform where every layer of the agent runtime: cognition, memory & state, action, and the trust & governance plane — runs on a single platform built on the Lakehouse. This unified architecture enables the following: no data copies between systems, no governance gaps between tools, and a single platform team to operate, reducing TCO and accelerating time-to-value versus a multi-vendor assembly:
- Lakeflow Connect for real-time CDC ingestion from ERP/WMS/TMS, with Structured Streaming for continuous processing and anomaly detection — no separate Flink deployment required
- Unity Catalog for fine-grained access control, lineage, and governance across every tool, skill, and data asset the agent touches
- Unity AI Gateway for FM APIs with guardrails, cost controls, and rate limits enforced at a single choke point
- Model Training and Serving for end-to-end management of the Anomaly Detection models
- Custom-built agents with the Supervisor pattern for managed orchestration and background-mode execution
- Genie for human planner exploration alongside agent-driven UC function tools
- Vector Search for RAG-based policy retrieval from unstructured documents
- Lakebase with low-latency, full ACID transactions for the durable task queue, agent memory, checkpointing, and decision logging — operational state and analytics on one copy of the data
- Databricks Apps for the always-on worker and the human SME interface — scorecard review, parameter tuning, exception handling
- MLflow for agent tracing, evaluation, and performance monitoring
