I met similar issues, I doubt it's related to spark udf. In my case, I load a sklearn model and use spark udf to speed up model predict, it raised similar error. However if I don't use udf, it could work. An exception was thrown from the Python work...
