Flex nodes

[fkseki]

Databricks REST API docs of instance pool editing states it supports the node_type_flexibility object but I couldn't make it work. I get a 200 response but when getting the instance pool information the alternate node type is not present. Does anyone know how to edit an instance pool in order to define an alternate node type?

Here's what I tried passing on the request body

{

"instance_pool_id": "<instance_pool_id>",

"instance_pool_name": "DeadPool",

"min_idle_instances": 0,

"node_type_id": "Standard_D8as_v5",

"node_type_flexibility": {

    "alternate_node_type_ids": [

      "Standard_D8as_v4", "Standard_D8as_v3"

    ]

  },

"idle_instance_autotermination_minutes": 5,

"enable_elastic_disk": true,

"azure_attributes": {

"availability": "ON_DEMAND_AZURE"

}

}

If the instance pool is created with the node_type_flexibility, it's configuration comes on the request, but I'm still unable to edit it's list of alternate node types

Am I doing something wrong or is the documentation wrong?

[Pat]

hi @fkseki ,
this looks interesting, I found this page: https://learn.microsoft.com/en-us/azure/databricks/compute/flexible-node-type-instances

it seems like your instances are not compatible for the alternate nodes:

Your Primary Listed Compatibles

Standard_D8as_v5

Standard_D8s_v5, D8s_v4, D8as_v6, DC8es_v5, Standard_F8as_v6

[fkseki]

Thanks for replying, @Pat !

I didn't see that page before, thanks for sharing!

What I think is really weird is the fact that they state that for an instance to be compatible it must be the same CPU architecture and D8as_v5 is AMD while D8s_v5 is Intel. I'll try tomorrow and share the results

[fkseki]

Well, I redid the test by first creating a pool with Standard_D8s_v5 and then trying to edit it adding the flex node Standard_D8as_v5, which is a real necessity for us and is a viable option according to documentation. However, got the same result. Response 200 OK, but the pool doesn't have the flex node configured.

I also tried creating a pool from scratch using the REST API and passing an instance not compatible. It created the pool and the flex node configured, with said instance. After trying to create the pool again passing the flex node "foo", I got a INVALID_PARAMETER_VALUE and a message that the valid value is actually a white list of any instance. It doesn't check if the flex node is compatible to the instance on the pool. So we're back at the start, if anyone has any other suggestion

[fkseki · Accepted solution]

Ok, just to provide an answer for anyone facing the same issue:

Support informed me that although the REST API documentation accepts the node_type_flexibility parameter, it fails to mention that this attribute cannot be modified when editing an existing instance pool.

If you want to use this attribute, node_type_flexibility must be set at the time of creation; it is immutable once the pool is created.

[Pat]

Thanks for sharing the answer!