If someone comes across this post - as per documentation, library/package installation can be Notebook-scoped. Thus, in order to overcome the limitation described in the initial post instead we are experimenting with Notebook tasks whose only responsibility it is to install the custom library using %pip install followed by a call to main() of module which contains the actual processing logic.

I am surprised that running PySpark jobs packaged as .whl in isolation is not something that Databricks provides out of the box. Ways to do so via for instance packaged virtual environments are described in PySpark's documentation and I would have expected Databricks to handle .whl tasks in such a way without the user having to worry about one job interfering with another.

Regards,

David