I agree with you regarding Scala UDFs being helpful. Given that I personally know very little Scala, and that Python UDFs are much less performant than SQL UDFs and Scala UDFs, I prefer to use SQL UDFs when possible. It takes me 20 times longer or wo...
This doesn't answer the question, but apparently someone asked a similar question on StackOverflow too and also got zero replies https://stackoverflow.com/questions/75010883/error-when-passing-spark-sql-higher-order-functions-lambda-variable-to-custo...
