Databricks notebook is connecting ADLS Gen2 using Service principal authentication and the setup is working fine. The notebook is able to read/write files to ADLS Gen2. However, occasionally, we are seeing below error in the production environment:
java.net.SocketTimeoutException at java.net.SocketInputStream.socketRead
HttpRequestFailure: 0,err=,appendpos=,cid=b165d74b-8177-4e24-9226-c4ce42dda733,rid=,connMs=0,sendMs=0,recvMs=0,sent=0,recv=0,method=HEAD,url=https://zXXXXXX01.dfs.core.windows.net/clusters/user/cxxxxxy/rxx-inc-json/xx/x/7xxx4/xxx-xxxx-xxxx.j...
------
Cluster Details:
------
"spark_version": "10.4.x-scala2.12",
"node_type_id": "Standard_D8s_v5",
"driver_node_type_id": "Standard_D16ds_v5"
---
Any help you can provide would be greatly appreciated.
Thanks
