gaponte
New Contributor III

Have you installed the Java/Scala package as well, or just the Python wrapper? It's possible to install just the latter, but the code won't actually run successfully when called if you haven't installed the Java/Scala package.

The issue seems to be with how the SynapseML developers packaged their jar. Yesterday they instructed me that the most recent version of SynapseML works against Spark 3.2, so I downgraded to the Databricks ML Runtime 10.4 and got it installed and running.

I just learned that a slightly older version of SynapseML purportedly works on Spark 3.3, so I'm going to re-upgrade back to Databricks runtime 12.1 and see if that runs.

For anyone else who may be attempting to train LightGBM models in a distributed way, the issue is documented here.