AgentBricks - Multi Agent Supervisor Issue

[Abhishek_Patel]

Hi, I am doing POC using AgentBricks and using Multi Agent Supervisor. I have configured 2 subagents.

1. Genie Space

2. UC Function to call LLM

3. UC Function to plot charts

However, the agent is able to access and use only Genie space agent and not using either of the UC functions. I am using a single user to conduct these POC and provided all the necessary access on all relevant objects. I even tried a very basic functions and that too was not detected and used by the agent in spite of being configured correctly. Currently I am stuck and seeking any advise on what could be the probable issue. Any help is much appreciated.

Note : I am using UC function to access LLMs as Databricks has not yet enabled using LLM endpoints in Agent Endpoint subagent.

Thanks.

[Krishna_S]

Hi @Abhishek_Patel 

I am just curious why you are calling LLM inside a UC function. It should always be the other way around. Also, the failure could be the UC function to access LLMs, as Databricks has not yet enabled using LLM endpoints in the Agent Endpoint subagent. But can you create simple UC functions to see if the calling is working? This functionality has just been added to the Agent Bricks, so I would test it with simple UC functions. Please let me know, and share the error and logs if you are still facing the issue with simple UC functions.

[Abhishek_Patel]

Thanks for the response Krishna. I am calling LLM from UC function as direct LLM endpoint functionality is till not enabled in AgentBricks. Are you suggesting that even that won't work as AgentBricks is still not integrated with LLMs? Also, I did create a simple UC function but that too is not recognized by Multi Agent Supervisor even after configuring it successfully within Multi Agent Supervisor. My use case is to primarily use Genie space to answer questions related to Data and augment/enhance answers with LLM when needed and hence I want to integrate the same. Hope this clarifies.

[KaushalVachhani]

Hi @Abhishek_Patel , Could you please share a minimal code setup that can help us reproduce the behavior?

For context, I tried replicating the issue based on your description, but I couldn't. I set up a multi-agent supervisor with two agents handling music and track sales-related queries:

• Genie agent configured with the digital media sales database.
• UC function agent that wraps a REST API call to an LLM for any music-related question.

Below is how my function definition looks.

Based on the agent descriptions, it either triggers a Python function that internally calls an LLM (as shown in the first snapshot below) or invokes a Genie space to generate the response (as shown in the second snapshot below).

[Abhishek_Patel]

Thanks @KaushalVachhani for such a detailed suggestion. Here is what I have configured. Due to org policy I am not able to upload any screenshot.

function to call llm

=========

%sql
CREATE OR REPLACE FUNCTION ask_llm(
question STRING
)
RETURNS STRING
LANGUAGE PYTHON
AS $$
import requests

endpoint = "<ENDPOINT>"
token = "<TOEKN>"

headers = {
"Authorization": f"Bearer {token}",
"Content-Type": "application/json"
}
data = {
"messages": [
{"role": "user", "content": question}
]
}

response = requests.post(endpoint, headers=headers, json=data)
if response.status_code == 200:
return response.json().get("choices", [{}])[0].get("message", {}).get("content", "")
else:
return f"Error: {response.text}"
$$

And this the configuration of Multi Agent Supervisor . 1st Subagent Genie followed by above mentioned function as 2nd. When I ask the agent "What all UC functions you can use?" It give below answer.

Available Function:

agent-fins: This is a Genie space agent that can access risk data and answer risk-related questions. It converts queries into SQL commands and executes them on the data warehouse.

It shows only Genie Agent. And when I ask the question that needs llm, e.g. Give me top 5 FINS org with highest Risk monitoring, it respond back stating that data does not have the information required instead of using llm and listing top 5 org based on the public data. I hope this helps.

[Abhishek_Patel]

Hi @KaushalVachhani , can you please share your multi agent setup? After properly qualifying function name with catalog and schema, multi agent is able to access the function. However, for some reason, my supervisor agent, agent-fins, which is a Genie space is not passing the question to the llm function agent. I tried multiple prompts and instructions, but it is not using it. The details on your agent setup would be very helpful. Also, any suggestions on setting access and privileges.

[KaushalVachhani]

@Abhishek_Patel , My setup is very lightweight. Here is how my MAS setup looks.

I already shared my function definition earlier. Hope this helps!

[Abhishek_Patel]

HI @KaushalVachhani , I have been doing multiple changes, and got to the point where UC functions are visible to the agent however it is not able to use as it expects agent-fins, Genie agent, to be the supervisor and mandatorily pass question to UC function through it. Multi agent is not able to directly call this UC function. And since the question is outside of the data available, supervisor agent, agent-fins, is outright rejecting the question and not passing it on to UC function. Here is what is says.

Agent Response:

=============

For general knowledge questions like "Show me the bank with the most number of branches," I should route directly to function-fins-llm-response since it's designed to handle questions outside the scope of the risk data.

However, I notice that function-fins-llm-response is not available as a direct function call in my current tool set. I only have access to:

1. agent-fins (for risk data queries)
2. transfer_to_agent-fins

According to my instructions, function-fins-llm-response should be used for general knowledge questions, but it appears this function is not currently accessible to me directly. The agent-fins should be able to pass questions to function-fins-llm-response internally, but as we saw, it's not doing that for general knowledge questions.

You're correct that I should have followed the proper routing for general knowledge questions. The system seems to have a gap where general knowledge questions that should go to function-fins-llm-response are not being handled as intended by the current agent setup.

==================

Perhaps this might give some clue as why functions are mandatorily being called from Genie agent and why Genie agent is not passing the questions to UC function in spite of a direct instructions to do so.

[KaushalVachhani]

Hi @Abhishek_Patel, I understand your Org policy, but it would be challenging to provide further suggestions without more information about your MAS setup. If you could provide a minimal setup that can be used to reproduce the behavior with an open-source dataset, that would be helpful.

If you have access to Databricks support, please feel free to raise a support ticket with them to expedite.

[Abhishek_Patel]

@Krishna_S @KaushalVachhani btw I am using DB Express instance to conduct this POC. Do you suspect any limitations with DB Express? I saw several info stating that it is not, but need confirmation.

[Krishna_S]

One last test: Can you completely remove the Genie Spaces and keep UC functions, and then test it in the playground? Let me know how it goes.

[Krishna_S]

Also, as you can see on the top right of the attached image, it creates an MLFlow experiment.

Click on that MLflow experiment, and check the traces and let me know what it shows, as you can see here it shows the tool is being called in the trace.

[Abhishek_Patel]

Hi @Krishna_S , since I was compiling the function without full qualifier name, agent was not able to locate it. After recompiling with the full qualifier name, agent is able to locate the function but not able to access the same though Genie agent which seems to be the supervisor agent. I have posted more details on above thread. And in mlflow experiment, it never shows UC function call.

And an agent with just this function is working fine, but not working when used with Genie agent.