Lakeflow Community Connector Issue - ModuleNotFoundError: No module named 'databricks.labs'

ChristianRRL
Honored Contributor

Tuesday

Hi there, I have what should be a somewhat straightforward implementation of a Custom Community Connector. I've tried to Dry run a few times but am encountering PYTHON.MODULE_NOT_FOUND_ERROR

Below is some context about my setup:

  • Source Relative path: lakeflow-community-connectors\src\databricks\labs\community_connector\sources\graph_sharepoint_lists
    • ChristianRRL_1-1784652953771.png
  • Ingest Relative path: lakeflow-community-connectors\src\ingest.ipynb
    • ChristianRRL_0-1784652943308.png

I suspect it "should" work out of the box, but for good measure I've tried adding the sys path to the */src, but I'm still getting the same error.

import sys
sys.path.insert(0, "/Workspace/Users/***@***.com/lakeflow_connect/graph_sharepoint_lists_pipeline_test1/src")

 Anyone here familiar with Lakeflow Community Connectors that can help steer me in the right direction?

0 Kudos

iyashk-DB
Databricks Employee
Databricks Employee

Tuesday

The sys.path you added is pointing to the wrong directory. For from databricks.labs.community_connector import ... to resolve, Python needs to find the src/ folder at the root of the cloned repo (the one that contains the databricks/ namespace directory), not a subdirectory inside it.

The clearest fix is to install the package directly in your notebook before running anything else:

%pip install -e /Workspace/Users/<your-user>/lakeflow-community-connectors

Restart the Python process after that (dbutils.library.restartPython()), and the imports should resolve.

If you'd rather keep the sys.path approach, the path needs to be the src/ directory at the top of the cloned repo, not a subdirectory of it:

import sys
sys.path.insert(0, "/Workspace/Users/<your-user>/lakeflow-community-connectors/src")

One more thing worth knowing: the pyproject.toml in that repo explicitly excludes databricks.labs.community_connector.sources.* from the installed wheel (the exclude in [tool.setuptools.packages.find]). So the connector source code (like graph_sharepoint_lists) isn't bundled into the package and is expected to be accessed directly from the workspace file path. As long as the src/ root is on your path, that's fine since Python will walk down into the sources directory from there.

If you're deploying through the official CLI (community-connector create_pipeline), it sets up a Git-backed DLT pipeline where the library paths are configured automatically. That flow avoids the manual path setup entirely.

binlogreader
New Contributor
In response to iyashk-DB

Tuesday

I am starting to explore community connector. Do they need to be installed in notebook, or can I package it as a wheel and (a) upload on a volume? (b) install on the cluster?

0 Kudos

ChristianRRL
Honored Contributor
In response to iyashk-DB

Tuesday

Hi @iyashk-DB , what you're saying mostly makes sense, but the initial `sys.path.insert` I posted should be accurate. For example, I show how my Workspace is setup, and you can trace same path:

/Workspace/Users/***@***.com/lakeflow_connect/graph_sharepoint_lists_pipeline_test1/src

ChristianRRL_0-1784665472255.png

And for some reason, what I posted earlier doesn't work:

import sys
sys.path.insert(0, "/Workspace/Users/***@***.com/lakeflow_connect/graph_sharepoint_lists_pipeline_test1/src")

I also tried your suggestion to pip install the package, but it doesn't seem to be working. Since this is a Lakeflow Connect pipeline, I don't know if it actually lets me run commands like that on a cell. Instead, I have to either (A) Dry run the pipeline, (B) Run pipeline, or (C) cry.

ChristianRRL_1-1784665729346.png

I'm sure there's something missing here, but as far as the ingest notebook + repo structure, it "seems" fine to me!

0 Kudos

iyashk-DB
Databricks Employee
Databricks Employee

yesterday

the sys.path approach won’t fix this for Lakeflow Connect pipelines. During a dry run, Databricks imports databricks.labs.community_connector before your ingest notebook runs, using only the pipeline’s Environment dependencies.

Fix: Pipeline Settings → Pipeline environment → Edit environment → add:

-e /Workspace/Users/<you>/lakeflow_connect/graph_sharepoint_lists_pipeline_test1

Use the repo root that contains pyproject.toml and src/ (not src/ itself). Then dry run again.

Also confirm the full package exists under src/databricks/labs/community_connector/ — if you only copied your connector source + ingest.ipynb without the framework files, the import will still fail.

%pip install in the ingest notebook isn’t supported for pipelines; dependencies must be declared in Environment settings. See: https://docs.databricks.com/aws/en/ldp/developer/external-dependencies

View solution in original post