multi agent Genie AI app in serving endpoint fails with PERMISSION ERROR on Databricks tabls.

[jpradeepkumar]

I have deployed simple multi agent genie app on serving endpoint and it always fails PERMISSION error on tables accessed by Genie agent, however the same app works perfectly when tested from notebook. not sure what am i missing here. I checked all the permissions for my user and i am the workspace admin and I have given all privileges to all the tables but still fails at serving review chat app. I have followed steps mentioned in this notebook to create app.

https://docs.databricks.com/aws/en/notebooks/source/generative-ai/dspy/dspy-multiagent-genie.html

Please help here. 

Genie query failed with error: {'error': \"PERMISSION_DENIED: Failed to fetch tables for the space. Please resolve these errors to continue: No access to table 'xxxxx.xxxxx.xxx'. Code: 42.\", 'type': 'TABLES_MISSING_EXCEPTION'}"

[davidmorton · Accepted solution]

Are you using OBO authentication within the app, and passing that to the MAS endpoint?

https://docs.databricks.com/aws/en/generative-ai/agent-framework/agent-authentication

[jpradeepkumar]

@davidmorton 
Tried with OBO authentication as per above link you have shared, but still end up with this exception and I see Databricks Free edition at this moment for experimenting.

{"error_code": "BAD_REQUEST", "message": "Encountered an unexpected error while evaluating the model. Verify that the input is compatible with the model for inference. Error 'model_serving_user_credentials auth: Unable to detect credentials for user authorization. This error has two common causes: (1) Improper OBO configuration - ensure you logged your model with a UserAuthPolicy and that the 'Agent Framework: On-Behalf-Of-User Authorization' preview is enabled in your workspace. (2) WorkspaceClient instantiation outside of predict()/predict_stream() - ensure you instantiate the WorkspaceClient inside your predict() or predict_stream() function, not at model-loading time. See https://docs.databricks.com/aws/en/generative-ai/agent-framework/authenticate-on-behalf-of-user for details. If the issue persists, contact Databricks Support.'", "stack_trace": "Traceback (most recent call last):\n File \"/opt/conda/envs/mlflow-env/lib/python3.12/site-packages/databricks/sdk/config.py\", line 691, in init_auth\n self._header_factory = self._credentials_strategy(self)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/opt/conda/envs/mlflow-env/lib/python3.12/site-packages/databricks_ai_bridge/model_serving_obo_credential_strategy.py\", line 181, in __call__\n raise ValueError(\nValueError: Unable to detect credentials for user authorization. This error has two common causes: (1) Improper OBO configuration - ensure you logged your model with a UserAuthPolicy and that the 'Agent Framework: On-Behalf-Of-User Authorization' preview is enabled in your workspace. (2) WorkspaceClient instantiation outside of predict()/predict_stream() - ensure you instantiate the WorkspaceClient inside your predict() or predict_stream() function, not at model-loading time.

[jpradeepkumar · Accepted solution]

hi @davidmorton 
thanks for responding, I am only using databricks_token for my authentication when tested via notebook. Can you help what changes needs to be done as per below notebook to enable access to tables when model deployed in serving endpoint.

https://docs.databricks.com/aws/en/notebooks/source/generative-ai/dspy/dspy-multiagent-genie.html