Users should be able to have a single Python environment setup with a single set of Python dependencies specified (in pyproject.toml or similar) and installed, and alternately point their code at either a local or remote Spark cluster simply by changing the URL they pass to `DatabricksSession.builder.remote(...)`.

As far as I can tell, this is not possible. Databricks Connect does not work with a local, open source Spark Connect server (i.e. what you get when you run `sbin/start-connect-server.sh`). And open source Spark Connect does not work with a remote Databricks cluster. And as others have pointed out, installing `databricks-connect` and `pyspark` side-by-side yields a broken Python environment.

No one wants to have a Databricks cluster running 24/7 just so they can run their tests quickly. And having multiple Python environments just to handle these incompatibilities means being forced to abandon modern Python packaging tooling like Poetry and going back to manually wrangling venvs.

Is there a design reason Databricks cannot simply enable Databricks Connect to work with open source Spark Connect servers?