Hello Community!

writing to you with the question about what are the best way to run spark unit tests in databricks. Currently we have a set of notebooks which are responsible for doing the operations on the data (joins, merging etc.).
Of course to do not keep everything in the notebooks we have separate directory for python functions. Very often they use spark code. For now the only way to to test those functions is to mock Spark but this is not how we would like to keep it since we mocking the outputs so real Spark behavior is skipped. The problem is that to run spark tests we need databricks environment. Maybe another important info is that we use serverless for the calculations since we do not want to wait for the cluster to wake up. 

Do you have any suggestions how to write some nice spark tests and be able to run them in Databricks env?

Thanks a lot!