I want to run some unit tests on my code, but Databricks can't seem to handle running formal unit testing libraries due to the lack of command line. From Googling, it appears it's possible to run notebooks and such from IntelliJ if using Scala, rather than using the Databricks interface. Is there a way to do this similarly with a Python IDE, e.g. PyC https://9apps.ooo/harm? I'd like to avoid running Spark locally in lieu of our already existing clusters.
(I could also be wrong about not being able to run pytest or similar properly in Databricks... just couldn't get it to work, anyway, so I'm open to suggestions on that too.)
