Governance to restrict compute creation

[RicksDB]

Hi,

Cluster policies used to be an easy way to handle governance on computes. However, more and more, there seem to be no way to control many new compute features within the platform. 

We currently have this issue for model serving endpoints and vector search. Non admin users can create them and we have no control over this. The same is true for jobs and DLT.

Am I missing something or is there really no way to control this. If not, is there anything in the roadmap to address those issues?

Thanks!

[Alberto_Umana]

Hi @RicksDB,

Thanks for your question! there is no direct way to prevent non-admin users from creating model serving endpoints, vector search or DLT... however please note that to execute them it would required compute which can be indeed restricted to users.

[RicksDB]

Thanks Alberto,

Clusters are required for jobs and dlt. However, vector search and model serving do not not have any controls similar to cluster policies . Therefore, it doesn't seem possible to control cost by restricting them or at the very least, force a minimal configuration to restrict the DBU usage.

Is there something similar to cluster policies for those endpoints?

Thanks

[Alberto_Umana]

There are no direct equivalents to cluster policies for vector search and model serving endpoints at the moment, to control its usage it's done by regular permissions, but at creation there is still not restriction. I will raise an internal feature request for this.

[RicksDB]

Thanks for the internal feature request.

Meanwhile, is there any Databricks recommended "monitoring scripts/feature" that we can use "as-is" in order to delete automatically endpoints that are not created by workspace admins? (Without a 24 hours delay)

We intend to offer the platform "as a service" to many teams within the company. However, by doing so, we pretty much give a blank check to them and serving can be quite costly if they choose GPUs by error. Ideally, we would not depend on a homemade script for such an important feature.

Thanks!

[nskiran]

If you are looking to restrict end users to create certain cluster configuration only, you can do so by using databricks APIs. Through python and Databricks API, you can specify what kind of cluster configurations are allowed and also restrict users to specific AWS/AZURE/GCP cloud storages through role based access controls.

Ex: By using API, you can restrict users to select DBR 11.x version and specific cluster type etc. You can refer to below link for more details

https://docs.databricks.com/api/workspace/clusters/create

[RicksDB]

As far as I know, it only works for clusters used by all-purpose , jobs, dlt and SQL workloads.

The new computes such as vector search endpoints and model serving can be automated using APIs but cannot be blocked in the UI (ex: any user when workspace access can create multiple 64 dbu hour endpoints). For now, we are mitigating that risk by telling people not to create them and use budgets monitor but it doesn't work well since tags are not mandatory either and you get noticed after 24 hours.

Users with only SQL access are fine since those menus are blocked.