Hi ML folks,

We are using Databricks to train deep learning models. The code, however, has a complex structure of classes. This would work fine in a perfect bug-free world like Alice in Wonderland.

Debugging in Databricks is awkward. We ended up doing all the development on local machines and when the code is "mature" we start playing with Databricks + MLFlow to train the model. We use Azure not only for databricks, but also for data. However, we ended up having a "security hole" with this approach. IT staff want to remove this type of permission from local machines, which will create a challenge for the ML team...

Google search didn't do much to find a good reference that works as a guideline for doing "complex" python-based model development and training in databricks. Any suggestion is welcome!

Cheers