Could Not Connect to ADLS Gen2 Using ABFSS

[tap]

I'm new to Databricks, not sure what can I do about this issue. I run a simple comment to list all file paths but get SSLHandshakeException.

Is there any way to resolve this?

 

The full error message

ExecutionError              Traceback (most recent call last)

<command-2100345877892203> in <module>

----> 1 dbutils.fs.ls("abfss://[account name].dfs.core.windows.net/")

/databricks/python_shell/dbruntime/dbutils.py in f_with_exception_handling(*args, **kwargs)

  317           exc.__context__ = None

  318           exc.__cause__ = None

--> 319           raise exc

  320 

  321       return f_with_exception_handling

ExecutionError: An error occurred while calling z:com.databricks.backend.daemon.dbutils.FSUtils.ls.

: Status code: -1 error code: null error message: InvalidAbfsRestOperationExceptionjavax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching [account name].dfs.core.windows.net found.

at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.executeHttpOperation(AbfsRestOperation.java:316)

at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:226)

at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.AbfsClient.listPath(AbfsClient.java:234)

at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:915)

at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:877)

at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.listStatus(AzureBlobFileSystem.java:463)

at com.databricks.backend.daemon.dbutils.FSUtils$.$anonfun$ls$1(DBUtilsCore.scala:154)

at com.databricks.backend.daemon.dbutils.FSUtils$.withFsSafetyCheck(DBUtilsCore.scala:91)

at com.databricks.backend.daemon.dbutils.FSUtils$.ls(DBUtilsCore.scala:153)

at com.databricks.backend.daemon.dbutils.FSUtils.ls(DBUtilsCore.scala)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)

at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)

at py4j.Gateway.invoke(Gateway.java:295)

at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)

at py4j.commands.CallCommand.execute(CallCommand.java:79)

at py4j.GatewayConnection.run(GatewayConnection.java:251)

at java.lang.Thread.run(Thread.java:748)

Caused by: javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching [account name].dfs.core.windows.net found.

at sun.security.ssl.Alert.createSSLException(Alert.java:131)

at sun.security.ssl.TransportContext.fatal(TransportContext.java:348)

at sun.security.ssl.TransportContext.fatal(TransportContext.java:291)

at sun.security.ssl.TransportContext.fatal(TransportContext.java:286)

at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)

at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)

at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)

at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377)

at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)

at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422)

at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182)

at sun.security.ssl.SSLTransport.decode(SSLTransport.java:156)

at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1418)

at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1324)

at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:439)

at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:410)

at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)

at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:197)

at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570)

at sun.net.www.protocol.http.HttpURLConnection.access$200(HttpURLConnection.java:92)

at sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1490)

at sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1488)

at java.security.AccessController.doPrivileged(Native Method)

at java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:784)

at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1487)

at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)

at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352)

at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.AbfsHttpOperation.processResponse(AbfsHttpOperation.java:348)

at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.executeHttpOperation(AbfsRestOperation.java:293)

Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching [account name].dfs.core.windows.net found.

at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:230)

at sun.security.util.HostnameChecker.match(HostnameChecker.java:106)

at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:457)

at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:417)

at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:230)

at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129)

at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638)

Thanks in advance!

[Prabakar]

Hi @suet pooi tan​  could you please check the authentication values that you are using to authenticate the storage? I believe there should be a problem with the authentication method.

https://docs.microsoft.com/en-us/azure/databricks/data/data-sources/azure/adls-gen2/

[tap]

Thanks for your advice, @Prabakar Ammeappin​ !

Let me double-check the requirements.

I'm wondering if it possible the SSL handshake exception caused by networking or firewall settings but I'm not sure are my thought is correct.

[Hubert-Dudek]

please try also:

%sh

nslookup your adls domain

as maybe your dns entry is incorrect (it have to be taken from endpoints in azure accounts)

It can be also routing problem or you are not authenticated as @Prabakar Ammeappin​ said.

You can try to mount adls directory - it is easier that way.

[jose_gonzalez]

I think is more like an authentication issue than a routing issue, but it might be a good idea to double check just to make sure.

[Hubert-Dudek]

I know but I had similar problem with private link as here name is redacted so we can not see what type of link is used (azure gives you private link but in fact you need to put normal link and check is it resolving correctly to private ip, additionally both private link to dfs and blob have to be created for the same resources as otherwise it is not working)

[Prabakar]

Hello @suet pooi tan​ have you verified the configuration? Have you checked the DNS information using nslookup? Have you tried mounting the storage and checked? Also, do you have any firewall setup?

[tap]

Apologies for the delayed response. We have confirmed the issue caused by the missing private endpoint in the Azure portal. Lastly, I appreciate the time you all spend and thank you for being so helpful😀

[Anonymous]

@suet pooi tan​ - Thank you for letting us know. 😎