Databricks notebook is configured with ADLS gen2 using Service principal authentication and is able to read/write files to ADLS gen2. However, occasionally, we are seeing below errors in the production environment:
java.net.SocketTimeoutException at java.net.SocketInputStream.socketRead
HttpRequestFailure: 0,err=,appendpos=,cid=b165d74b-8177-4e24-9226-c4ce42dda733,rid=,connMs=0,sendMs=0,recvMs=0,sent=0,recv=0,method=HEAD,url=https://zXXXXXX01.dfs.core.windows.net/clusters/user/cxxxxxy/rxx-inc-json/xx/x/7xxx4/xxx-xxxx-xxxx.j...
----
Spark_Version: 0.4.x-scala2.12
---
Are there any timeout settings that can be tweaked to resolve this issue?
Thanks
