topic maven libraries installation issue on 11.3/12.2 LTS in Data Engineering

maven libraries installation issue on 11.3/12.2 LTS

Ravikumashi — Thu, 17 Aug 2023 17:48:57 GMT

we've encountered issue while attempting to install Maven libraries on Databricks clusters version 11.3 LTS. Specifically, we are encountering SSL handshake errors during the installation process. It's worth noting that these same libraries install without any issues on Databricks versions up to 10.4 LTS.

We would greatly appreciate your insights and assistance in resolving this matter.

one example for such maven library is 'com.microsoft.azure:applicationinsights-core:2.6.1'

@azuredatabrick @azuredatabrick

Re: maven libraries installation issue on 11.3/12.2 LTS

Ravikumashi — Fri, 18 Aug 2023 10:59:04 GMT

Thank you for your reply.

We have tried all of this and also init script we use has no issue and looks like something has changed from 11.0 LTS and above

Re: maven libraries installation issue on 11.3/12.2 LTS

Ravikumashi — Fri, 18 Aug 2023 10:55:49 GMT

as per the behavioural changes in 11.3 and above, maven libraries now resolve in your data plane by default when you install libraries on a cluster. You cluster must have access to maven central. Alternatively, you can revert to the previous behavior by setting the Spark configuration property: spark.databricks.libraries.enableMavenResolution false

with this flag set to false, i was able to get the maven libraries installed as like in earlier version.

I thought this behavior is same in older version, it would be great if someone can clarify how it works in 10.4 LTS and below