the hacky solution above is meant to be used only while developing my own python module - this way I can avoid packaging a whl, deploying to the cluster, restarting the cluster and even restarting the notebook interpreter.

I agree that it is not suited for production. For that I would use either a whl ref in the workflow file or just prepare a docker image.