Is following an option for you? 

https://www.databricks.com/blog/announcing-storage-optimized-endpoints-vector-search

Also, good to have your SA address this further as underlying issue may be somewhere else - model serving performance for instance.