maven libraries installation issue on 11.3/12.2 LTS

[Ravikumashi]

we've encountered issue while attempting to install Maven libraries on Databricks clusters version 11.3 LTS. Specifically, we are encountering SSL handshake errors during the installation process. It's worth noting that these same libraries install without any issues on Databricks versions up to 10.4 LTS.

We would greatly appreciate your insights and assistance in resolving this matter.

one example for such maven library is  'com.microsoft.azure:applicationinsights-core:2.6.1'

@azuredatabrick @azuredatabrick 

[Kaniz_Fatma]

Hi @Ravikumashi, The SSL handshake errors you're encountering during the Maven library installation on Databricks clusters version 11.3 LTS might be due to transient Maven issues or conflicts with existing libraries. There are no specific solutions provided in the sources for SSL handshake errors.

However, there are a few general suggestions that might help:

1. Retry the installation: As per the first source, the issue might resolve itself if you try again later, as it might be a transient Maven issue.

2. Check for carriage return characters: According to the second source, a carriage return character at the end of one or more lines in the init script could cause issues. You can remove these by running dos2unix on the file after uploading it to your Databricks workspace.

3. Check for conflicting libraries: The third source suggests that installation failures can occur if the library tries to install already installed dependencies. You can resolve this by excluding the dependencies before starting the installation. Unfortunately, providing a more precise solution without more specific information about the error and your setup is challenging. If the issue persists, I recommend contacting Databricks support by filing a support ticket for further assistance.

Sources:
- [Libraries failing due to transient Maven issue](https://kb.databricks.com/libraries/library-fail-transient-maven)
- [Init script fails to download Maven JAR](https://kb.databricks.com/libraries/init-script-fail-download-maven)
- [Failed to install Elasticsearch via Maven](https://kb.databricks.com/libraries/failed-to-install-elasticsearch-via-maven)

[Ravikumashi]

Thank you for your reply.

We have tried all of this and also init script we use has no issue and looks like something has changed from 11.0 LTS and above

[Ravikumashi]

as per the behavioural changes in 11.3 and above, maven libraries now resolve in your data plane by default when you install libraries on a cluster. You cluster must have access to maven central. Alternatively, you can revert to the previous behavior by setting the Spark configuration property: spark.databricks.libraries.enableMavenResolution false

with this flag set to false, i was able to get the maven libraries installed as like in earlier version.

I thought this behavior is same in older version, it would be great if someone can clarify how it works in 10.4 LTS and below