Designing Hierarchical Multi-Agent Systems on Databricks with Supervisor Agent

[Dwaipayan2001]

When does one AI agent stop being enough?

Many enterprise AI applications begin with a simple architecture:

User → LLM → Answer

As applications become more capable, the pattern usually evolves into: 

User → Agent → Tools → Answer

Now the agent can decide when to query data, search documents, call an API, execute a function, or use another specialised capability. This works well—until the number of capabilities starts growing.

Imagine an enterprise assistant that needs to answer questions involving:

• structured business data
• internal policies and documentation
• customer support systems
• operational APIs
• external services
• multi-step investigations

The obvious solution might be to give a single agent access to everything.

But eventually, the problem is no longer simply "Can the agent call tools?"

The problem becomes:

How should the system decide which capability is responsible for which part of a complex request?

That is where multi-agent orchestration becomes useful.

And when the number of agents and domains continues to grow, a flat multi-agent architecture can eventually face the same scaling problem that affected the original single agent.

This is where a hierarchical multi-agent architecture becomes interesting.

In this article, I'll walk through how such an architecture can be designed using Databricks Supervisor Agent, with domain-level supervisors coordinating specialised Databricks agents and tools.

The problem: more capabilities create a larger orchestration problem

Consider a straightforward business question:

What was our total revenue last quarter?

A single analytical capability can handle this.

Now consider a more realistic enterprise question:

Which product had the highest return rate last quarter, what does our warranty policy say about that product, and are there any unresolved customer issues related to it?

This request spans multiple domains.

1. Analytics

Determine which product has the highest return rate.

2. Enterprise knowledge

Find the warranty policy relevant to that product.

3. Operations

Retrieve unresolved customer issues associated with it. A single agent could theoretically have access to all these capabilities.

However, as the toolset grows, the agent increasingly needs to reason about:

• Which capability should I use?
• Which tool should I call first?
• Can multiple capabilities run independently?
• Does the output from one agent become input for another?
• Which results should be combined?
• When is the investigation complete?

At that point, we are no longer solving a simple tool-calling problem.

We are solving an orchestration problem.

From a single agent to a team of agents

The first step is often specialization.

Instead of one agent handling everything, we might create:

Supervisor
    |
    +── Analytics Agent
    +── Knowledge Agent
    +── Operations Agent

This is already an improvement.

Each specialist can have a clearer responsibility:

• The Analytics Agent focuses on structured data.
• The Knowledge Agent focuses on documents and policies.
• The Operations Agent focuses on enterprise systems and actions.

But what happens when the organization has many more domains?

Root Supervisor
    |
    +── Finance
    +── Sales
    +── Customer Service
    +── Operations
    +── HR
    +── Legal
    +── Research
    +── ...

The root supervisor now needs to understand every specialist's purpose and decide how to route requests across all of them. We have effectively moved the complexity upward. That is where hierarchy can help.

Why this architecture maps well to Databricks

Databricks Supervisor Agent is designed to coordinate specialized agents and tools across complex tasks.

Depending on the use case, a supervisor can coordinate capabilities such as:

• Genie Agents
• Knowledge Assistant endpoints
• model serving endpoints
• Unity Catalog functions
• MCP servers
• custom agents
• other Supervisor Agents

This is particularly interesting for hierarchical architectures because a Supervisor Agent can itself become part of a larger orchestration structure.

That gives us a natural separation between:

Enterprise-level orchestration

and:

Domain-level orchestration

Databricks also provides governance and access-control mechanisms around the underlying agents and tools, so orchestration does not mean bypassing the permissions associated with the resources being used.

Designing the hierarchy

A common mistake in multi-agent systems is creating multiple agents simply because a multi-agent architecture sounds more advanced.

More agents do not automatically produce a better system. The goal should be to create clear boundaries of responsibility.

For this example, let's use three domain supervisors.

1. Analytics Supervisor

This supervisor handles questions involving:

• revenue
• sales
• orders
• products
• customers
• KPIs
• trends
• etc...

A possible architecture is:

Analytics Supervisor
          |
         v
   Genie Agent
          |
         v
  Structured Data

The important design principle is that the Analytics Supervisor does not necessarily need to generate SQL itself.

Its responsibility is to determine:

• whether the request belongs to the analytics domain,
• what information needs to be retrieved,
• whether additional analytical investigation is required,
• how the analytical result should be returned to the higher-level supervisor.

The underlying Genie capability can handle the structured-data interaction.

2. Knowledge Supervisor

This supervisor handles enterprise knowledge such as:

• company policies
• product documentation
• internal procedures
• FAQs
• warranty information
• operational guidelines

A possible structure is:

Knowledge Supervisor
          |
          v
 Knowledge Assistant
          |
          v
 Enterprise Documents

This keeps document-oriented reasoning separate from structured-data analysis.

That separation becomes useful when a request requires both numerical evidence and policy or documentation context.

3. Operations Supervisor

This supervisor handles interactions with operational systems.

For example:

• customer support tickets
• inventory
• product status
• enterprise APIs
• external services

A possible structure could be:

Operations Supervisor
          |
          v
      MCP
          |
    +-----+-----+------+
    |         |        |          |
Tickets Inventory Status APIs

The MCP layer can expose domain-specific capabilities to the supervisor while keeping the operational integrations behind a clear boundary.

For example:

get_open_tickets()
get_inventory()
get_product_status()

The role of the Enterprise Supervisor

The Enterprise Supervisor sits above the domain supervisors.

Its responsibility should be fundamentally different from that of the specialists.

A good Enterprise Supervisor should:

1. Understand the user's overall request.
2. Identify the domains involved.
3. Break the request into delegable tasks.
4. Route tasks to the appropriate domain supervisor.
5. Pass relevant context between domains when necessary.
6. Coordinate dependent and independent work.
7. Decide whether further investigation is required.
8. Synthesise the final answer.

Conceptually:

                    User Question
                          |
                          v
                Enterprise Supervisor
                          |
          +---------------+---------------+
          |                      |                      |
          v                    v                     v
    Analytics         Knowledge       Operations
    Supervisor        Supervisor      Supervisor

This is the central architectural separation:

The enterprise layer manages domains. Domain layers manage specialized capabilities.

Building the architecture incrementally

A practical implementation should not begin by creating the entire hierarchy at once.

Build and validate each layer separately.

Step 1: Build the specialist capabilities

Start with the underlying capabilities:

• a Genie Agent for structured analytics,
• a Knowledge Assistant for enterprise documents,
• an MCP-based capability for operational systems.

Before introducing orchestration, make sure each capability works independently.

For example:

Analytics question → Genie
Policy question → Knowledge Assistant
Operational question → MCP tool

If the specialists are unreliable individually, adding a supervisor will not solve the underlying problem.

Step 2: Create domain supervisors

Next, create supervisors for each domain.

For example:

Analytics Supervisor
    → analytics capabilities only

Knowledge Supervisor
    → knowledge capabilities only

Operations Supervisor
    → operational capabilities only

The objective is to keep each supervisor's decision space focused. Avoid giving every domain supervisor access to every tool. Specialization is the point.

Step 3: Create the Enterprise Supervisor

Now add the domain supervisors as capabilities of the Enterprise Supervisor.

At this layer, the descriptions become extremely important.

The Enterprise Supervisor should understand clearly when each domain is appropriate.

For example:

Analytics Supervisor

Use for questions requiring numerical analysis, business metrics, trends, comparisons, aggregations, KPIs, sales, revenue, orders, products, or other structured business data.

Knowledge Supervisor

Use for questions about company policies, product documentation, procedures, FAQs, warranty information, and internal knowledge.

Operations Supervisor

Use for questions requiring information or actions from operational systems, support tickets, inventory, product status, or external enterprise services.

These boundaries should overlap as little as possible.

Step 4: Design for dependencies, not just routing

This is one improvement I would strongly recommend when building a real hierarchical system.

Not every task is simply:

Route → Execute → Return

Some tasks have dependencies.

For example:

Find product with highest return rate
              |
              v
Use product as input
              |
      +-------+-------+
      |               |
      v               v
Warranty lookup   Ticket lookup
      |               |
      +-------+-------+
              |
              v
        Final synthesis

The first analytical result becomes context for the next two domain tasks. This means evaluation should test not only whether the system selected the correct agent, but also whether it correctly passed information between agents.

How should a hierarchical agent be evaluated?

I would evaluate the architecture across several dimensions.

1. Routing accuracy

Did the Enterprise Supervisor select the correct domain?

Correct domain selections
-------------------------
Total routing decisions

2. Delegation accuracy

Once inside a domain, did the domain supervisor select the correct specialist capability?

For example:

Analytics Supervisor
        |
        +── Correctly used Genie
        |
        +── Did not incorrectly invoke an operational tool

3. Dependency accuracy

Did the system correctly pass information between agents?

For example:

Analytics result:
XYZ-Phone Pro

↓

Knowledge request:
Find the warranty policy for XYZ-Phone Pro

This becomes especially important for multi-step investigations.

4. Tool selection

Was the correct underlying tool invoked?

Correct routing to the Analytics Supervisor is not enough if the Analytics Supervisor then chooses the wrong capability.

5. Unnecessary calls

Did the system invoke agents or tools that were not required?

A good multi-agent architecture should not simply maximize collaboration. It should use the minimum necessary capabilities to complete the task reliably.

6. Final answer quality

Finally:

• Is the answer correct?
• Is it grounded in the results returned by the relevant systems?
• Does it combine evidence correctly?
• Are contradictions between agents handled properly?
• Does the final answer answer the user's actual question?

When does hierarchy become useful?

A hierarchical architecture becomes more attractive when:

• domains contain multiple specialized capabilities,
• each domain requires its own orchestration logic,
• the number of agents is growing,
• the root supervisor is becoming overloaded,
• different teams own different capabilities,
• workflows naturally contain multiple levels of responsibility,
• tasks require cross-domain dependencies.

The architecture should follow the problem.

Do not build a hierarchy because multi-agent systems look more sophisticated. Build one when hierarchy reduces the orchestration complexity of the problem.

 

Final thoughts

The future of enterprise AI is unlikely to be a single massive agent with an endlessly growing list of tools. But that does not mean every problem requires a large hierarchy of agents either.

The important architectural question is:

Where should orchestration responsibility live as the number of capabilities grows?

Databricks provides a useful foundation for exploring this question through Supervisor Agent and its ability to coordinate specialized agents and tools across domains.

My approach would be to start simple:

Single Agent

Then move to:

Flat Multi-Agent

And introduce:

Hierarchical Multi-Agent

only when the growing number of domains and capabilities creates an orchestration problem that hierarchy actually helps solve. The best architecture is not the one with the most agents. It is the one that gives every layer a clear responsibility.

AI assistance disclosure: This article was AI-assisted for editorial restructuring and language refinement. The architecture, examples, technical claims, and final content were reviewed and verified by the author.