To anyone struggling: This was resolved by changing the URL to the endpoint of the rest api documentation url instead of the endpoint invocation url. https://docs.databricks.com/api/azure/workspace/servingendpoints/query"/serving-endpoints/{name}/inv...
I managed to retrieve the access token but when i call the served endpoint i get the same error as the user above. my code is below. As for the SP in databricks workspace, it was given the manage permission on the served endpoint. Any suggestions?