cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Unable to install SynapseML on clusters

gaponte
New Contributor III

I would like to run a distributed training using LightGBM but I cannot install SynapseML. I have tried doing so on a few different clusters (note: our clusters are running on AWS, not sure if that matters. Also, I am running the Databricks ML Runtime v12.1) and all fail with the same error.

Here are the steps I am taking:

1) Navigate to the Libraries tab on the page of the cluster that I would like to install SynapseML on.

2) Click the "Install new" button to bring up the Install Library modal.

3) Populate that modal by selecting Maven, then setting the Coordinates to com.microsoft.azure:synapseml_2.12:0.10.2 and the Repository to https://mmlspark.azureedge.net/maven . This is per the instructions on the SynapseML documentation site (see the Databricks section here).

4) Start cluster

After a little while, the cluster starts but the installation of SynapseML errors out, with the following error text:

Library installation attempted on the driver node of cluster 0206-190349-ms8qnkwe and failed. Please refer to the following error message to fix the library or contact Databricks support. Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Message: java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File file:/local_disk0/tmp/clusterWideResolutionDir/maven/ivy/jars/com.microsoft.azure_onnx-protobuf_2.12-0.9.1.jar does not exist

Please let me know how I can successfully install SynapseML and unblock myself. Thanks so much

5 REPLIES 5

Anonymous
Not applicable

LightGBM is already installed on the ml runtime

https://docs.databricks.com/release-notes/runtime/12.1ml.html

gaponte
New Contributor III

Yes it is, and with what is installed I am able to train a model on a single instance the "normal way", i.e. what one could do locally on their laptop.

However, I would like to run a distributed training, which requires the installation of SynapseML (unless you have evidence that that is NOT required to accomplish that).

Anonymous
Not applicable

You should be able to install it with PyPi. I just tried it and was able to install the 0.10.2 release

gaponte
New Contributor III

Have you installed the Java/Scala package as well, or just the Python wrapper? It's possible to install just the latter, but the code won't actually run successfully when called if you haven't installed the Java/Scala package.

The issue seems to be with how the SynapseML developers packaged their jar. Yesterday they instructed me that the most recent version of SynapseML works against Spark 3.2, so I downgraded to the Databricks ML Runtime 10.4 and got it installed and running.

I just learned that a slightly older version of SynapseML purportedly works on Spark 3.3, so I'm going to re-upgrade back to Databricks runtime 12.1 and see if that runs.

For anyone else who may be attempting to train LightGBM models in a distributed way, the issue is documented here.

Anonymous
Not applicable

Hi @Greg Aponte​ 

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future.

Thank you for your participation and let us know if you need any further assistance! 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group