Databricks model serving endpoint returns 403 Unauthorized access to workspace when using service

[ashfire]

I deployed a simple Iris model in Databricks Model Serving and exposed it as an endpoint. I’m trying to query the endpoint using a service principal. I can successfully fetch the access token with the following databricks_token() function:

def databricks_token(): 
    token_url = f"https://accounts.cloud.databricks.com/oidc/accounts/{MY_ACCOUNT_ID}/v1/token"
    scope = "all-apis"

    data = {
        "grant_type": "client_credentials",
        "client_id": CLIENT_ID,
        "client_secret": CLIENT_SECRET,
        "scope": scope,
    }

    response = requests.post(token_url, data=data)
    token_data = response.json()
    access_token = token_data["access_token"]
    return access_token

Then I try to query the endpoint using score_model():

def score_model():
    url = f"https://{WORKSPACE_HOST}.cloud.databricks.com/serving-endpoints/{MODEL_SERVING_ENDPOINT_NAME}/invocations"
    headers = {'Authorization': f'Bearer {databricks_token()}', 'Content-Type': 'application/json'}
    data_json = json.dumps(data, allow_nan=True)
    response = requests.request(method='POST', headers=headers, url=url, data=data_json)
    if response.status_code != 200:
        raise Exception(f'Request failed with status {response.status_code}, {response.text}')
    return response.json()

print(score_model())

But the call fails with: Exception: Request failed with status 403, {"error_code":"403","message":"Unauthorized access to workspace: xxxxxxxxxx"}

In the Databricks UI, the serving endpoint already has the permission “All workspace users can query”.

What am I missing to allow a service principal to query the model serving endpoint? Do I need to assign additional workspace or service principal permissions beyond the endpoint-level access?
Note that the route optimization is not enabled here.

[szymon_dybczak]

Hi @ashfire ,

It seems that your service principal is getting a an access token at Account level, but model serving endpoints live at the workspace level. In databricks you have 2 separate API: 

- Account level API

- Workspace Level

As you can see at below screen - the serving endpoint lives at workspace level so it expects access token generated at that level.

Here you can read more details about the difference between those two and how to generate workspace level access token

Authorizing access to Azure Databricks resources - Azure Databricks | Microsoft Learn

[ashfire]

Hi @szymon_dybczak, Thanks for your comment.
One of the admins in this workspace tried using the token generated via client id and secret, and were able to successfully get a response from the serving endpoint using this same above mentioned code.

Could this be related to specific permissions required for regular users? I would really appreciate it if you could elaborate or share an example to help resolve this issue. Thanks again for your help and guidance.

[szymon_dybczak]

HI @ashfire ,

After second read of documentation I think the endpoint you choose should work (with scope="all-apis"). According to docs it will let you call both REST API , so sorry for confusion in previous message.

The reason it works out of the box for workspace administrator is because workspace admin has all necessary permission in workspace. 
But for any other user I think you need to grant them explicitly. Could you check if your Service Principal has appropriate permission on your model serving endpoint?

Manage model serving endpoints | Databricks on AWS

Also, please check if your Service Principal has been added to the workspace:

And lastly, check if he has workspace access entitelment enabled: