topic Re: Issue with Maven Dependency Resolution in Databricks (json-smart) in Data Engineering

Issue with Maven Dependency Resolution in Databricks (json-smart)

VictorFerron — Wed, 12 Feb 2025 11:03:52 GMT

Hi,
I'm experiencing issues installing a library in my Databricks cluster due to a Maven dependency resolution error. Specifically, when trying to install or use libraries that depend on net.minidev:json-smart, I get the following error:

Library installation attempted on the driver node and failed. Error code: ERROR_MAVEN_LIBRARY_RESOLUTION, error message: Library resolution failed because unresolved dependency: net.minidev:json-smart:[1.3.1,2.3]: not found.

After researching, it seems that there is an issue with the Maven metadata for json-smart, causing some versions to be unresolved. I've tried:

Manually specifying json-smart:2.4.8 in the cluster library settings.
Adding alternative repositories (https://repo1.maven.org/maven2/, https://jitpack.io).
Cleaning the cluster library cache and restarting.
Excluding json-smart from Azure SDK dependencies to force a specific version.

None of these have resolved the issue. Has anyone encountered this problem in Databricks? Any suggestions on how to force a specific version or work around the Maven metadata issue?

Thanks in advance!

Re: Issue with Maven Dependency Resolution in Databricks (json-smart)

MLob — Wed, 12 Feb 2025 14:04:13 GMT

Same problem, same tests, same results. Currently working on it, support is very appreciated!

Re: Issue with Maven Dependency Resolution in Databricks (json-smart)

VictorFerron — Wed, 12 Feb 2025 14:04:18 GMT

To avoid this issue, I manually excluded the json-smart dependency when installing affected libraries and forced a stable version instead.
If you're facing this issue, this could be a viable solution until the json-smart team fixes the metadata in Maven Central.

Re: Issue with Maven Dependency Resolution in Databricks (json-smart)

MLob — Wed, 12 Feb 2025 14:13:30 GMT

Thanks VitorFerron, exlcuding json-smart dependency worked for installing affected library with Maven (azure-eventhubs-spark_2.12:2.3.22 in my case).

When you say force a stable version instead, do you mean explicitly install json-smart library on cluster with Maven (in my case net.minidev:json-smart:2.3 for example)?

Re: Issue with Maven Dependency Resolution in Databricks (json-smart)

VictorFerron — Wed, 12 Feb 2025 14:22:06 GMT

You're welcome

Yes, by ‘stable version,’ I mean using a Spark connector version that allows excluding json-smart. Installing an older version manually (like 2.3) may still cause issues, so if excluding it works, I’d recommend sticking with that approach.