I would like to run a distributed training using LightGBM but I cannot install SynapseML. I have tried doing so on a few different clusters (note: our clusters are running on AWS, not sure if that matters. Also, I am running the Databricks ML Runtime v12.1) and all fail with the same error.
Here are the steps I am taking:
1) Navigate to the Libraries tab on the page of the cluster that I would like to install SynapseML on.
2) Click the "Install new" button to bring up the Install Library modal.
3) Populate that modal by selecting Maven, then setting the Coordinates to com.microsoft.azure:synapseml_2.12:0.10.2 and the Repository to https://mmlspark.azureedge.net/maven . This is per the instructions on the SynapseML documentation site (see the Databricks section here).
4) Start cluster
After a little while, the cluster starts but the installation of SynapseML errors out, with the following error text:
Library installation attempted on the driver node of cluster 0206-190349-ms8qnkwe and failed. Please refer to the following error message to fix the library or contact Databricks support. Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Message: java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File file:/local_disk0/tmp/clusterWideResolutionDir/maven/ivy/jars/com.microsoft.azure_onnx-protobuf_2.12-0.9.1.jar does not exist
Please let me know how I can successfully install SynapseML and unblock myself. Thanks so much