After further investigation, It reproduces slightly differently on single user mode.
Single user mode: runs forever
Shared: gives the above message

I've determined that there was a corner case in the dataset which lead to UDF never returning. I am am assuming that on shared the udf runner is killed by a watchdog and that is why the error message is so cryptic (rather than "Timeout" type error)