I have run into similar issues before. Make sure DNS is working properly!If the cluster is up you can verify dns with:!dig somehostnameOr enable cluster logs and add some debug (routing / dns) statements in start-up script. They are stored on dbfs
Can you confirm this is a known issue?Running into same issue, example to test in 1 cell.# using Arrow fails on HighConcurrency-cluster with PassThrough in runtime 10.4 (and 10.5 and 11.0)
spark.conf.set("spark.sql.execution.arrow.pyspark.enabled",...
