Could Not Connect to ADLS Gen2 Using ABFSS

tap — Mon, 06 Dec 2021 15:21:28 GMT

I'm new to Databricks, not sure what can I do about this issue. I run a simple comment to list all file paths but get SSLHandshakeException.

Is there any way to resolve this?

The full error message

ExecutionError Traceback (most recent call last)

<command-2100345877892203> in <module>

----> 1 dbutils.fs.ls("abfss://[account name].dfs.core.windows.net/")

/databricks/python_shell/dbruntime/dbutils.py in f_with_exception_handling(*args, **kwargs)

317 exc.__context__ = None

318 exc.__cause__ = None

--> 319 raise exc

320

321 return f_with_exception_handling

ExecutionError: An error occurred while calling z:com.databricks.backend.daemon.dbutils.FSUtils.ls.

: Status code: -1 error code: null error message: InvalidAbfsRestOperationExceptionjavax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching [account name].dfs.core.windows.net found.

at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.executeHttpOperation(AbfsRestOperation.java:316)

at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:226)

at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.AbfsClient.listPath(AbfsClient.java:234)

at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:915)

at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:877)

at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.listStatus(AzureBlobFileSystem.java:463)

at com.databricks.backend.daemon.dbutils.FSUtils$.$anonfun$ls$1(DBUtilsCore.scala:154)

at com.databricks.backend.daemon.dbutils.FSUtils$.withFsSafetyCheck(DBUtilsCore.scala:91)

at com.databricks.backend.daemon.dbutils.FSUtils$.ls(DBUtilsCore.scala:153)

at com.databricks.backend.daemon.dbutils.FSUtils.ls(DBUtilsCore.scala)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)

at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)

at py4j.Gateway.invoke(Gateway.java:295)

at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)

at py4j.commands.CallCommand.execute(CallCommand.java:79)

at py4j.GatewayConnection.run(GatewayConnection.java:251)

at java.lang.Thread.run(Thread.java:748)

Caused by: javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching [account name].dfs.core.windows.net found.

at sun.security.ssl.Alert.createSSLException(Alert.java:131)

at sun.security.ssl.TransportContext.fatal(TransportContext.java:348)

at sun.security.ssl.TransportContext.fatal(TransportContext.java:291)

at sun.security.ssl.TransportContext.fatal(TransportContext.java:286)

at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)

at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)

at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)

at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377)

at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)

at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422)

at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182)

at sun.security.ssl.SSLTransport.decode(SSLTransport.java:156)

at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1418)

at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1324)

at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:439)

at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:410)

at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)

at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:197)

at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570)

at sun.net.www.protocol.http.HttpURLConnection.access$200(HttpURLConnection.java:92)

at sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1490)

at sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1488)

at java.security.AccessController.doPrivileged(Native Method)

at java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:784)

at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1487)

at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)

at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352)

at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.AbfsHttpOperation.processResponse(AbfsHttpOperation.java:348)

at shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.executeHttpOperation(AbfsRestOperation.java:293)

Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching [account name].dfs.core.windows.net found.

at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:230)

at sun.security.util.HostnameChecker.match(HostnameChecker.java:106)

at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:457)

at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:417)

at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:230)

at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129)

at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638)

Thanks in advance!

Re: Could Not Connect to ADLS Gen2 Using ABFSS

Prabakar — Mon, 06 Dec 2021 15:33:10 GMT

Hi @suet pooi tan could you please check the authentication values that you are using to authenticate the storage? I believe there should be a problem with the authentication method.

https://docs.microsoft.com/en-us/azure/databricks/data/data-sources/azure/adls-gen2/

Re: Could Not Connect to ADLS Gen2 Using ABFSS

tap — Mon, 06 Dec 2021 16:17:37 GMT

Thanks for your advice, @Prabakar Ammeappin !

Let me double-check the requirements.

I'm wondering if it possible the SSL handshake exception caused by networking or firewall settings but I'm not sure are my thought is correct.

Re: Could Not Connect to ADLS Gen2 Using ABFSS

Hubert-Dudek — Mon, 06 Dec 2021 16:42:12 GMT

please try also:

%sh

nslookup your adls domain

as maybe your dns entry is incorrect (it have to be taken from endpoints in azure accounts)

It can be also routing problem or you are not authenticated as @Prabakar Ammeappin said.

You can try to mount adls directory - it is easier that way.

Re: Could Not Connect to ADLS Gen2 Using ABFSS

jose_gonzalez — Tue, 07 Dec 2021 00:10:21 GMT

I think is more like an authentication issue than a routing issue, but it might be a good idea to double check just to make sure.

Re: Could Not Connect to ADLS Gen2 Using ABFSS

Hubert-Dudek — Tue, 07 Dec 2021 10:11:10 GMT

I know but I had similar problem with private link as here name is redacted so we can not see what type of link is used (azure gives you private link but in fact you need to put normal link and check is it resolving correctly to private ip, additionally both private link to dfs and blob have to be created for the same resources as otherwise it is not working)

Re: Could Not Connect to ADLS Gen2 Using ABFSS

Prabakar — Fri, 10 Dec 2021 10:08:45 GMT

Hello @suet pooi tan have you verified the configuration? Have you checked the DNS information using nslookup? Have you tried mounting the storage and checked? Also, do you have any firewall setup?

Re: Could Not Connect to ADLS Gen2 Using ABFSS

tap — Fri, 10 Dec 2021 19:00:46 GMT

Apologies for the delayed response. We have confirmed the issue caused by the missing private endpoint in the Azure portal. Lastly, I appreciate the time you all spend and thank you for being so helpful😀

Re: Could Not Connect to ADLS Gen2 Using ABFSS

Anonymous — Fri, 10 Dec 2021 20:02:09 GMT

@suet pooi tan - Thank you for letting us know. 😎

topic Re: Could Not Connect to ADLS Gen2 Using ABFSS in Data Engineering

Could Not Connect to ADLS Gen2 Using ABFSS

Re: Could Not Connect to ADLS Gen2 Using ABFSS

Re: Could Not Connect to ADLS Gen2 Using ABFSS

Re: Could Not Connect to ADLS Gen2 Using ABFSS

Re: Could Not Connect to ADLS Gen2 Using ABFSS

Re: Could Not Connect to ADLS Gen2 Using ABFSS

Re: Could Not Connect to ADLS Gen2 Using ABFSS

Re: Could Not Connect to ADLS Gen2 Using ABFSS

Re: Could Not Connect to ADLS Gen2 Using ABFSS