You can use the MLflow client (in various language specific SDKs) to download model artifacts. For example, see here: https://docs.databricks.com/en/mlflow/models.html#download-model-artifacts

We leverage this pattern to serve models in our K8s stack for similar reasons to what you stated (existing clusters, cost / performance, network latency benefits, etc.). I've also used Databricks Model Serving extensively, including in the original "CPU only on Spark clusters" version, and I much prefer the self-managed route if it's something your team has capacity to support.