topic Re: Real-time model serving and monitoring on Databricks at scale in Data Engineering

Real-time model serving and monitoring on Databricks at scale

Maverick1 — Fri, 10 Sep 2021 05:53:42 GMT

How to deploy real-time model on databricks at scale? Right now, The model serving is very limited to 20 requests per second. Also, There are no model monitoring framework/graphs like the one's provided with AzureML or Sagemaker frameworks.

Re: Real-time model serving and monitoring on Databricks at scale

-werners- — Fri, 10 Sep 2021 06:34:04 GMT

You might wanna look into MLFlow.

But as far as the deployment of models goes, MLFlow only does local REST APIs afaik.

Added to that you can also deploy to AzureML or Sagemaker.

Not sure what Databricks's plans are on the deployment part. I think they probably will go for out of the box integration with existing platforms, but Databricks people in here might shine a light on this.

Re: Real-time model serving and monitoring on Databricks at scale

-werners- — Fri, 10 Sep 2021 06:47:54 GMT

https://docs.databricks.com/applications/mlflow/model-serving.html

I guess I am not up to date anymore.

Re: Real-time model serving and monitoring on Databricks at scale

Maverick1 — Fri, 10 Sep 2021 08:48:27 GMT

@Werner Stinckens :

The real time capability is not yet scalable, but I have heard about an update to this in August product roadmap where databricks team have bifurcated the serving layer into 2 parts (Batch and Real-time). Not sure how much scalability is improved.

Also, There is nothing around model monitoring which is a big challenge while going to real-time model serving architecture.

Re: Real-time model serving and monitoring on Databricks at scale

-werners- — Fri, 10 Sep 2021 09:06:33 GMT

Agree.

Which is why at my company we look at Azure ML.

Re: Real-time model serving and monitoring on Databricks at scale

Sebastian — Fri, 10 Sep 2021 12:06:50 GMT

For real time serving probably you will have to look into container services with Kubernetes. And agree deployed through Azure ML

Re: Real-time model serving and monitoring on Databricks at scale

Anonymous — Sat, 11 Sep 2021 05:39:12 GMT

It's accurate that the current Databricks model serving product has limitations regarding scalability.

That being said, MLflow has built-in deployment tools for serving products, including cloud services and open source alternatives.

We do have improvements to both our serving product regarding scalability AND monitoring on our roadmap. Happy to discuss if you are interested!

Re: Real-time model serving and monitoring on Databricks at scale

Maverick1 — Mon, 27 Sep 2021 06:08:47 GMT

@Clemens Mewald : Thanks for your response.

I have heard about serving 2.0 . Would you be able to provide a rough timeline on when it will be available?

Does it include the below requirements:

multi-endpoint deployment (One model being deployed with multiple endpoints).
multi-region deployment (One model having end-points in different regions).
multi-model endpoints deployment (One end-point supporting multiple models)

Also, When will the apache airflow native integration would be available to use on databricks?

Re: Real-time model serving and monitoring on Databricks at scale

sean_owen — Sun, 10 Oct 2021 16:29:00 GMT

I believe the next update to serving will include 1, not 2 (this is still within a Databricks workspace in a region). I don't think multi-model endpoints are on the roadmap next.

How does Airflow integration relate?