This was due to a %run notebook command where the cluster could not locate the notebook. I was using a relative to root path from the github repo which did work when running the notebook book on a cluster but did not work when I put it into a job. However the task failed before it got to the notebook and did not say this was the problem at alll.