Getting error while installing applicationinsights

[shiv_DB25]

Library installation attempted on the driver node of cluster 0210-115502-3lo6gkwd and failed. Pip could not find a version that satisfies the requirement for the library. Please check your library version and dependencies. Error code: ERROR_NO_MATCHING_DISTRIBUTION, error message: org.apache.spark.SparkException: Process List(/bin/su, libraries, -c, bash /local_disk0/.ephemeral_nfs/cluster_libraries/python/python_start_clusterwide.sh /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/pip install 'azure-identity' --disable-pip-version-check) exited with code 1. WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))': /simple/azure-identity/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))': /simple/azure-identity/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))': /simple/azure-identity/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))': /simple/azure-identity/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))': /simple/azure-identity/
ERROR: Could not find a version that satisfies the requirement azure-identity (from versions: none)
ERROR: No matching distribution found for azure-identity

[shiv_DB25]

applicationinsights is library i am trying to install from cluster PyPi

[mark_ott]

The error indicates that Databricks could not install the applicationinsights or azure-identity libraries because pip could not find a matching distribution, and network connection attempts to the PyPI repository were repeatedly reset. This is commonly caused by network restrictions or firewall rules that prevent the cluster from accessing PyPI resources.

Common Causes

• The cluster cannot reach PyPI due to firewall, NSG, or VNet restrictions in Azure.

• The package name or version specified may be incorrect or unavailable on PyPI.

• Databricks is deployed in an environment with an ExpressRoute or private networking that restricts outbound internet access.

Troubleshooting Steps

• Confirm that your Azure Databricks workspace has access to the public internet, or ensure you have configured a private PyPI mirror or allowedlist for required domains (e.g., pypi.org, files.pythonhosted.org) for all cluster nodes.​

• Review the NSG and VNet rules to verify outbound access for protocol ports 443 (HTTPS). Even with NSG rules applied and ER lock removed, you may need explicit exceptions for Databricks clusters to access external PyPI repositories.​

• Check that the library name ("applicationinsights", "azure-identity") is spelled correctly and is available on PyPI.

• If using a custom PyPI repo or mirror (common in private networks), add the repository URL using the "--index-url" pip flag.

Recommended Solution

• If you are in a restricted environment (ExpressRoute, custom VNet), set up a private PyPI mirror with the needed packages, or allow access to PyPI URLs through your firewall.

• For immediate workaround, try installing the package by uploading the .whl file manually via the Databricks UI if you have access to the internet elsewhere.​

• Ensure your cluster is configured with appropriate permissions for outbound internet access, or adjust your NSG/firewall rules accordingly.

Useful Databricks Docs

Refer to Databricks documentation on library installation in restricted environments for network requirements and manual installation procedures. If you require detailed step-by-step instructions, the Databricks support site offers guides for both standard and custom library setup.

If these steps do not resolve your issue, consider posting the full error, along with details about your network setup, in the Databricks Community forums for more targeted troubleshooting from other users and support staff.​