topic Re: Can't Add Cluster-scoped Init Script to Model Serving Cluster in Machine Learning

Can't Add Cluster-scoped Init Script to Model Serving Cluster

anthonylavado — Tue, 07 Jun 2022 16:34:08 GMT

We're using Azure Databricks, and have a model that requires a WHL to be downloaded from a private address. As there is no public DNS record for the hostname, we found that we could use either a Global-Init Script to add the required hosts entry to an all-purpose cluster.

For a cluster created by serving a model, I set up an init script following the Azure documentation for a Cluster-scoped init script. Since this can't be specified by default when I start serving the model, I figured I would start it, then go to the settings for the job cluster and try to edit them. When I click on Edit, I put in the path to the script under the advanced options, but when I try to save the change, I get an error.

How can I get an init script to run with model serving clusters?

Re: Can't Add Cluster-scoped Init Script to Model Serving Cluster

anthonylavado — Thu, 09 Jun 2022 19:14:55 GMT

Hi @Kaniz Fatma (Databricks), That article is actually the one I was referring to in the middle of my post (re: cluster-scoped init script).

However, even following the same steps to create a script on the DBFS root, I still cannot add it to the cluster settings.

The exact steps that fail:

Go to cluster created by model serving
Click "Edit" for cluster
Open Advanced Options, go to Init Scripts
Input the script path with "dbfs:/" (example: "dbfs:/databricks/scripts/dns-add.sh")
Click Add, then "Confirm and Restart" at the top of the page

These steps are as seen in this section of the article: https://docs.microsoft.com/en-us/azure/databricks/clusters/init-scripts#configure-a-cluster-scoped-init-script-using-the-ui

When I click "Confirm and Restart", that's when an error appears: "Error: Cannot edit cluster ****-******-******** created by ModelServing".

I guess I have three questions then:

Is this a bug?
Is there a way to get this to work?
Is there any other easier/more consistent solution? This would require manually editing every model serving cluster that gets created.

Thanks!

-Anthony

Re: Can't Add Cluster-scoped Init Script to Model Serving Cluster

939772 — Mon, 21 Nov 2022 23:53:23 GMT

Has anyone had success with this? Trying to solve a resolve issue.