Had the following update from Databricks support.

"We can see the below error just before the repls started failing -

22/11/17 05:32:07 ERROR WSFSDriverManager$: Failed to get associated pid for WSFS

In the driver logs we could see several repls being initialized during that time. Going through similar scenarios with other customers in our backlogs we have seen reducing the concurrency helps mitigate the problem. Increasing the driver size will help as well since it will provide more cores for concurrent execution."

Still not convinced this gets to the root of the problem as everything seems stable now we have rolled clusters back to 10.4...