Hi all,
Thanks for the feedback and proposed ideas.
@szymon_dybczak  Your idea of relative imports work when the module is hosted in a child directory to the current running notebook. It does not work if we need to go up one or two directories and navigate from there.

The error in such case is "ImportError: attempted relative import with no known parent package". 
I will accept the solution proposed by @ck7007 
Practically speaking, I have to stick with things like "%run notebook" due to some other limitations.
Regarding the porposed options:

1. "spark.sparkContext.addPyFile" is also not supported on shared clusters
2. The CI templates I am bound to use does not allow tweaking init scripts easily
3. I tried the "exec" method before and it is working but feels a bit hacky and will get so complicated if we have modules depending on other modules and so on. It would become tricky easily.

If I have better control, I would have just used dedicated mode and off you go.

Thanks all.