1. Regarding unit testing of spark sql, I found it more easy and dynamic as we change any logic / join, unit test need not be changed. From pyspark point of view, the change in fn would directly mean changing the pytest of the fn. Need databricks expert opinion on this 2. For Complex Transformations, likee simple filter / window fn / aggregation, Wont SparkSQL be better than Pyspark ? (As we are importing pyspark.sql.function import window )
