Issue: Lakeflow Connect Microsoft Teams Community Connector - No module named 'databricks.labs'

ChristianRRL
Honored Contributor II

a month ago

Hi there, I'm trying to run a pretty barebones test here with community connectors. I'm trying out the "vanilla" Microsoft Teams connector by:

  • (A) going to the Data Ingestion tab in our workspace,
  • (B) selecting the Microsoft Teams community connector,
  • (C) creating a new basic connection based on what the Microsoft Teams connection requires (connection name, tenant id, client id, client secret), and
  • (D) setting up the ingestion pipeline (pipeline name, event log location, root path).

Very basic stuff.

Additionally, due to the way our network is setup, we can't leverage Serverless, so a small modification needed for us is to go to the pipeline settings (gear icon) > compute > untoggle Serverless. For simplicity, I'm keeping all the default non-serverless settings.

However, even with this extremely basic setup I'm seeing two issues when attempting to Dry run the microsoft teams ingestion pipeline (I have a workaround for the 1st, but not the 2nd issue):

1. Failed to load notebook due to error UNSUPPORTED_LANGUAGE

Failed to load notebook '/Users/***/lakeflow_connect/microsoft_teams_pipeline_test1/src/ingest.py'. Only SQL and Python notebooks are supported currently.
UNSUPPORTED_LANGUAGE
Failed to load  notebook '/Users/***/lakeflow_connect/microsoft_teams_pipeline_test1/src/ingest.py'. Only SQL and Python notebooks are supported currently.
  • The default generated ingest.py file is.. well it's a python file of course, but it's not being recognized.
  • I thought this might be due to "Default file format for notebooks" being set incorrectly. I updated it from Jupyter to Source, but I'm still seeing the issue.
  • My workaround here, is to create a new ingest file (notebook), copy the contents of the originally generated ingest.py file into the notebook, go to the pipeline settings (gear icon) > repoint the source code to the new ingest notebook, and run it again.

2. ModuleNotFoundError: No module named 'databricks.labs'

ChristianRRL_1-1784735725394.png

  • This error is pretty self-explanatory
  • There are a few attempted workarounds but none have worked so far:
    • Approach 1: import sys + sys.path.insert(0, ...)
    • Result 1: No change, same error
    • Approach 2: pip install -e ... + dbutils.library.restartPython()
    • Result 2: No change, same error
  • Hunch: I can't personally test this on Serverless, but maybe this works differently between Serverless and non-serverless (job compute I assume)

Not sure if there's something else, I would need to do here with the non-serverless compute configuration to make sure that this runs successfully. If not that, then I have no clue what may be going on.

Would heavily appreciate some assistance with this!

0 Kudos

iyashk-DB
Databricks Employee
Databricks Employee

a month ago

Both issues trace back to running on non-serverless compute. Community connectors were designed to run on serverless Spark Declarative Pipelines (SDP), and serverless is on by default in the CLI config for a reason.

The No module named 'databricks.labs' error happens because the pipeline relies on root_path (set to the repo's src/ directory in the workspace) to put databricks.labs.community_connector on the Python path. That root_path mechanism is specific to serverless DLT. On classic/non-serverless compute, that path isn't added automatically, so the import fails. Adding sys.path.insert in the notebook itself can work as a workaround, but only if you point it at the correct workspace path where the repo was cloned, like:

import sys
sys.path.insert(0, "/Workspace/Users/<you>/.lakeflow_community_connectors/<pipeline_name>/src")

Make sure that path matches where the pipeline's repo was actually checked out (you can find it in the pipeline's notebook location or the Workspace Files browser).

The UNSUPPORTED_LANGUAGE error for the generated Python notebook is a separate quirk. The Lakeflow Connect UI generates the pipeline notebook in a format that gets flagged as unsupported when serverless is off. Your workaround of manually recreating the notebook is the right call for now. The "Default file format for notebooks" setting doesn't affect auto-generated pipeline files, which is why changing it didn't help.

The cleanest path forward, if your network constraints allow it at all, is to use serverless compute. Community connectors just aren't tested or supported on classic clusters, and you'll keep running into friction. If you're blocked on serverless due to a private network/VPC requirement, you'd need to set up a private endpoint for serverless compute, which is a workspace-level network config, not something you can work around in the pipeline itself.

0 Kudos

ChristianRRL
Honored Contributor II
In response to iyashk-DB

4 weeks ago

Hey @iyashk-DB , cross-posting your solution from the related post:

Resolution Summary: The issue seems to be that since I have my pipeline compute settings using job compute (I toggled Serverless off because we can't currently use it in our organization), I was missing a couple of Pipeline environment Dependencies which are now allowing my pipeline to run!
/Workspace/Users/{username}/lakeflow_connect/microsoft_teams_pipeline_test1
/Workspace/Users/{username}/lakeflow_connect/microsoft_teams_pipeline_test1/src/databricks/labs/community_connector/sources/microsoft_teams

ChristianRRL_0-1784816934253.png

 

Once both dependencies were added, I was able to finally Dry run the ingest pipeline, troubleshoot some lingering configuration issues, and finally get the ingest pipeline to run:

ChristianRRL_1-1784816933942.png

 

Thanks again for the help here!

View solution in original post

0 Kudos