To expand on the same error "Could not reach driver of cluster XX" but different cause;

the reason in my case (ADF triggered databricks job which runs into this error) was a problem with a numpy library version, where solution is to downgrade the library on the cluster before run, e.g. "pip install numpy<2"