cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Real-time model serving and monitoring on Databricks at scale

Maverick1
Valued Contributor II

How to deploy real-time model on databricks at scale? Right now, The model serving is very limited to 20 requests per second. Also, There are no model monitoring framework/graphs like the one's provided with AzureML or Sagemaker frameworks.

1 ACCEPTED SOLUTION

Accepted Solutions

sean_owen
Databricks Employee
Databricks Employee

I believe the next update to serving will include 1, not 2 (this is still within a Databricks workspace in a region). I don't think multi-model endpoints are on the roadmap next.

How does Airflow integration relate?

View solution in original post

8 REPLIES 8

-werners-
Esteemed Contributor III

You might wanna look into MLFlow.

But as far as the deployment of models goes, MLFlow only does local REST APIs afaik.

Added to that you can also deploy to AzureML or Sagemaker.

Not sure what Databricks's plans are on the deployment part. I think they probably will go for out of the box integration with existing platforms, but Databricks people in here might shine a light on this.

-werners-
Esteemed Contributor III

Maverick1
Valued Contributor II

@Werner Stinckensโ€‹ :

The real time capability is not yet scalable, but I have heard about an update to this in August product roadmap where databricks team have bifurcated the serving layer into 2 parts (Batch and Real-time). Not sure how much scalability is improved.

Also, There is nothing around model monitoring which is a big challenge while going to real-time model serving architecture.

-werners-
Esteemed Contributor III

Agree.

Which is why at my company we look at Azure ML.

Sebastian
Contributor

For real time serving probably you will have to look into container services with Kubernetes. And agree deployed through Azure ML

Anonymous
Not applicable

It's accurate that the current Databricks model serving product has limitations regarding scalability.

That being said, MLflow has built-in deployment tools for serving products, including cloud services and open source alternatives.

We do have improvements to both our serving product regarding scalability AND monitoring on our roadmap. Happy to discuss if you are interested!

Maverick1
Valued Contributor II

@Clemens Mewaldโ€‹ : Thanks for your response.

I have heard about serving 2.0 . Would you be able to provide a rough timeline on when it will be available?

Does it include the below requirements:

  1. multi-endpoint deployment (One model being deployed with multiple endpoints).
  2. multi-region deployment (One model having end-points in different regions).
  3. multi-model endpoints deployment (One end-point supporting multiple models)

Also, When will the apache airflow native integration would be available to use on databricks?

sean_owen
Databricks Employee
Databricks Employee

I believe the next update to serving will include 1, not 2 (this is still within a Databricks workspace in a region). I don't think multi-model endpoints are on the roadmap next.

How does Airflow integration relate?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group