cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Endpoint creation without scale-to-zero

damselfly20
New Contributor II

Hi, I've got a question about deploying an endpoint for Llama 3.1 8b. The following code should create the endpoint without scale-to-zero. The endpoint is being created, but with scale-to-zero, although scale_to_zero_enabled is set to False. Instead of a boolean, I have also tried to pass the value as a string (both upper and lower case), but unfortunately this does not change the result. What do I have to change so that scale-to-zero is really deactivated?

 

cl = mlflow.deployments.get_deploy_client("databricks")
cl.create_endpoint(
   name="llama3_1_8b_instruct",
   config={
       "served_entities": [
           {
               "entity_name": "system.ai.meta_llama_v3_1_8b_instruct",
               "entity_version": "2",
               "max_provisioned_throughput": 12000,
               "scale_to_zero_enabled": False,
           }
       ],
       "traffic_config": {
           "routes": [
               {
                   "served_model_name": "meta_llama_v3_1_8b_instruct-2",
                   "traffic_percentage": "100",
               }
           ]
       },
   },
)

 

0 REPLIES 0

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group