Regarding complex transformations, we can use UDFs in SQL as well. So, we can still use sparkSQL, and delegate complex transformations into UDF.
