topic Re: Unable to install SynapseML on clusters in Machine Learning

Unable to install SynapseML on clusters

gaponte — Tue, 14 Feb 2023 18:00:03 GMT

I would like to run a distributed training using LightGBM but I cannot install SynapseML. I have tried doing so on a few different clusters (note: our clusters are running on AWS, not sure if that matters. Also, I am running the Databricks ML Runtime v12.1) and all fail with the same error.

Here are the steps I am taking:

1) Navigate to the Libraries tab on the page of the cluster that I would like to install SynapseML on.

2) Click the "Install new" button to bring up the Install Library modal.

3) Populate that modal by selecting Maven, then setting the Coordinates to com.microsoft.azure:synapseml_2.12:0.10.2 and the Repository to https://mmlspark.azureedge.net/maven . This is per the instructions on the SynapseML documentation site (see the Databricks section here).

4) Start cluster

After a little while, the cluster starts but the installation of SynapseML errors out, with the following error text:

Library installation attempted on the driver node of cluster 0206-190349-ms8qnkwe and failed. Please refer to the following error message to fix the library or contact Databricks support. Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Message: java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File file:/local_disk0/tmp/clusterWideResolutionDir/maven/ivy/jars/com.microsoft.azure_onnx-protobuf_2.12-0.9.1.jar does not exist

Please let me know how I can successfully install SynapseML and unblock myself. Thanks so much

Re: Unable to install SynapseML on clusters

Anonymous — Tue, 14 Feb 2023 18:09:25 GMT

LightGBM is already installed on the ml runtime

https://docs.databricks.com/release-notes/runtime/12.1ml.html

Re: Unable to install SynapseML on clusters

gaponte — Tue, 14 Feb 2023 18:13:22 GMT

Yes it is, and with what is installed I am able to train a model on a single instance the "normal way", i.e. what one could do locally on their laptop.

However, I would like to run a distributed training, which requires the installation of SynapseML (unless you have evidence that that is NOT required to accomplish that).

Re: Unable to install SynapseML on clusters

Anonymous — Wed, 15 Feb 2023 14:10:06 GMT

You should be able to install it with PyPi. I just tried it and was able to install the 0.10.2 release

Re: Unable to install SynapseML on clusters

gaponte — Wed, 15 Feb 2023 15:20:44 GMT

Have you installed the Java/Scala package as well, or just the Python wrapper? It's possible to install just the latter, but the code won't actually run successfully when called if you haven't installed the Java/Scala package.

The issue seems to be with how the SynapseML developers packaged their jar. Yesterday they instructed me that the most recent version of SynapseML works against Spark 3.2, so I downgraded to the Databricks ML Runtime 10.4 and got it installed and running.

I just learned that a slightly older version of SynapseML purportedly works on Spark 3.3, so I'm going to re-upgrade back to Databricks runtime 12.1 and see if that runs.

For anyone else who may be attempting to train LightGBM models in a distributed way, the issue is documented here.

Re: Unable to install SynapseML on clusters

Anonymous — Sat, 11 Mar 2023 03:21:30 GMT

Hi @Greg Aponte

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future.

Thank you for your participation and let us know if you need any further assistance!