Real-time model serving and monitoring on Databricks at scale

[Maverick1]

How to deploy real-time model on databricks at scale? Right now, The model serving is very limited to 20 requests per second. Also, There are no model monitoring framework/graphs like the one's provided with AzureML or Sagemaker frameworks.

[-werners-]

You might wanna look into MLFlow.

But as far as the deployment of models goes, MLFlow only does local REST APIs afaik.

Added to that you can also deploy to AzureML or Sagemaker.

Not sure what Databricks's plans are on the deployment part. I think they probably will go for out of the box integration with existing platforms, but Databricks people in here might shine a light on this.

[-werners-]

https://docs.databricks.com/applications/mlflow/model-serving.html

I guess I am not up to date anymore.

[Maverick1]

@Werner Stinckens​ :

The real time capability is not yet scalable, but I have heard about an update to this in August product roadmap where databricks team have bifurcated the serving layer into 2 parts (Batch and Real-time). Not sure how much scalability is improved.

Also, There is nothing around model monitoring which is a big challenge while going to real-time model serving architecture.

[-werners-]

Agree.

Which is why at my company we look at Azure ML.

[Sebastian]

For real time serving probably you will have to look into container services with Kubernetes. And agree deployed through Azure ML

[Anonymous]

It's accurate that the current Databricks model serving product has limitations regarding scalability.

That being said, MLflow has built-in deployment tools for serving products, including cloud services and open source alternatives.

We do have improvements to both our serving product regarding scalability AND monitoring on our roadmap. Happy to discuss if you are interested!

[Maverick1]

@Clemens Mewald​ : Thanks for your response.

I have heard about serving 2.0 . Would you be able to provide a rough timeline on when it will be available?

Does it include the below requirements:

1. multi-endpoint deployment (One model being deployed with multiple endpoints).
2. multi-region deployment (One model having end-points in different regions).
3. multi-model endpoints deployment (One end-point supporting multiple models)

Also, When will the apache airflow native integration would be available to use on databricks?

[sean_owen · Accepted solution]

I believe the next update to serving will include 1, not 2 (this is still within a Databricks workspace in a region). I don't think multi-model endpoints are on the roadmap next.

How does Airflow integration relate?