Newyn
Databricks Partner

That is not really what I want. What I want is to have a folder in a git repository connected to Databricks and that notebooks can search this folder or the repository for packages developed in-house. I know of the way to deploy them as wheel libraries both on clusters and in workspaces as well as how to connect them to Jobs.

It is much more convenient to be able to work in a proper IDE, then check in the change, check them out in Databricks and test them. By mistake I came across this in a project and the package was a folder in the root of the repository. When I ran tests, import found the folder in the repository. Very convenient. The tests resided in the same repository.

However, I have not fully understood or have had time to figure out how import could find this. There are numerous questions concerning this:

  1. Is the import search restricted to the same repository or can it go across to other repositories?
  2. Can import of this package deployed in this way be performed from a notebook in a workspace?
  3.  

I do not want to perform any ugly hacks changing sys.path inside the notebooks or the packages.