Lakeflow Connect - Community Custom Connector - How to troubleshoot source ingestion logic "live"

[ChristianRRL]

Hi there, I'm a bit confused about how I should be performing "live" troubleshooting with Lakeflow Connect with a Community Custom Connector I'm working on. Below is a simple example where I attempt to update the ingestion type from "cdc_with_deletes" to "snapshot" in the `{source}_schemas.py` file:

However, when I attempt to re-run the job (either regular or dry run), I see it's still running with the same prior configuration:

I assume that even after the changes are made, under the hood the python package isn't rebuilt. How should I go about making/testing changes more interactively in this case?

[ChristianRRL]

P.S. My configuration has the following required updates:

• Compute: Job Compute, due to serverless limitations at my company
  • 
• Pipeline environment > Dependencies:
  • /Workspace/Users/{username}/lakeflow_connect/microsoft_sharepoint_pipeline_test2
  • /Workspace/Users/{username}/lakeflow_connect/microsoft_sharepoint_pipeline_test2/src/databricks/labs/community_connector/sources/microsoft_sharepoint

[Niyojit]

are you developing this connector from the official Databricks Labs Community Connector template (using pyproject.toml and uv/poetry), or is this an internal connector with a custom project structure?

[ShamenParis]

Hi @ChristianRRL ,
Your compute cluster is caching the older version of your Python package in memory. Standard job runs or dry runs don't always force a rebuild or reload of workspace files.

I hope below steps will helps you,

1. Trigger a Full Refresh: Instead of a regular run or dry run, trigger a Full Refresh on the pipeline. This forces the engine to re-evaluate the code and pull the latest workspace files.

2. Use Editable Install: Ensure your pipeline dependencies are installing your custom package in editable mode by adding -e (e.g., pip install -e /Workspace/Users/...). This links directly to the source files rather than a static build.

3. Develop Interactively (Best for Live Testing): Testing via Job Compute is too slow for active development. Attach a standard Notebook to an All-Purpose cluster, run %load_ext autoreload followed by %autoreload 2, and test your connector logic there first. It will instantly pick up any changes you make to {source}_schemas.py without needing to rebuild.

[ChristianRRL]

Hi @ShamenParis , your suggestions mostly make sense, but unfortunately, I'm still struggling with the same issue. I updated the `ingestion_type` in both the {source}_schemas.py and the _generated_{source}_python_source.py files. Below are my attempts based on your suggestions:

1. Trigger a Full Refresh:
   • No difference
   • It's still showing up with the old `ingestion_type`
2. Use Editable Install:
   • ModuleNotFoundError: No module named 'databricks.labs.community_connector.pipeline'
   • Added `-e ` prior to my current dependencies but got above ERROR
3. Develop Interactively (Best for Live Testing):
   • This one doesn't make too much sense to me..
   • Even after running the commands, I'm not sure how a notebook connected to all-purpose compute is supposed to run the Lakeflow Connect pipeline logic.

Out of all these options, the 2nd one seems like it holds the most promise.. but I'm not sure how to get past the No module error.

[ShamenParis]

Hi @ChristianRRL

That makes total sense, and it's completely understandable why that interactive notebook suggestion felt out of place. Let's tackle that ModuleNotFoundError, because you are spot on—getting the editable install working is the key to bypassing the cache issue on Job Compute.

Fixing the ModuleNotFoundError (The Editable Install)

The error No module named 'databricks.labs.community_connector.pipeline' means Python is looking for the base framework files and cannot find them. Because Community Connectors are primarily designed to run on Serverless compute, forcing them to run on classic Job Compute requires very strict pathing.

Here is how to resolve it in your Pipeline Settings > Environment > Dependencies:

• Point to the Project Root: The -e flag must point to the absolute root directory of your cloned repository. This is the folder that contains your pyproject.toml file and the src/ folder. Do not point it to the src/ folder itself or your specific connector subfolder.

  • Use this: -e /Workspace/Users/{username}/lakeflow_connect/microsoft_sharepoint_pipeline_test2

• Verify the Framework Files Exist: Ensure the full base package actually exists in your workspace under src/databricks/labs/community_connector/. If you only copied your custom microsoft_sharepoint folder and ingest file without the rest of the Databricks Labs framework scaffolding, the pipeline will fail because the base classes are genuinely missing.
  
  

Clarifying the "Interactive Notebook" Suggestion

You are absolutely correct that you cannot run a full Lakeflow Connect / DLT pipeline in a standard all-purpose notebook.

The goal of that suggestion was strictly for testing your raw Python logic in isolation. For example, if you want to verify that your API calls or the schema definitions in {source}_schemas.py are functioning properly, you can import those specific Python classes into a standard notebook. Using %autoreload allows you to tweak the python files and test the data extraction instantly, without waiting for a job cluster to spin up. Once the extraction logic works perfectly in isolation, you can trust it will work when DLT wraps it in the pipeline.

Try adjusting that dependency path to the folder containing your pyproject.toml. That should force the Job Compute cluster to read your live files and correctly pick up the snapshot ingestion type update.

[ChristianRRL]

@ShamenParis AI response detected.

But in all seriousness, laser focusing on the 2nd one (ModuleNotFoundError), I mentioned that I already added `-e ` in my dependencies (see below):

Adding `-e ` is what is somehow leading to the ModuleNotFoundError. If I don't include `-e `, the pipeline Dry runs successfully, but will not update after I make logic changes to {source}_schema.py.

If I remove `-e ` from the root dependency, but keep it on the .../{source}/ dependency, I get a similar but different error:

For some reason, adding the `-e ` flag is leading to the above errors. If I exclude that flag (as I had been doing before), I get a successful dry run.. but the logic is not editable (my original problem).

[ShamenParis]

Hi @ChristianRRL 

Haha, you caught me! I was bouncing ideas off my AI assistant to try and unblock you.

It makes sense that the -e flag is throwing that ModuleNotFoundError—the Lakeflow/DLT sandbox clearly doesn't play well with the symlinks that editable installs rely on.

Since those initial suggestions didn't do the trick, I'm going to replicate this exact setup in my own environment today, test it out, and reply back to you once I find a concrete workaround!

[ChristianRRL]

Appreciate it in advance! For simplicity, I think this can be replicated with any Community Connector. E.g. I've replicated this same behavior with the `azure_devops` Community Connector.

Basically, follow these steps:

• Create a new Lakeflow Connect Community Connector Pipeline (e.g. with azure_devops)
• Pipeline Settings:
  • Compute > Untoggle Serverless
  • Pipeline environment > Dependencies:
    • -e /Workspace/Users/{user}/{pipeline_path}/
    • -e /Workspace/Users/{user}/{pipeline_path}/src/databricks/labs/community_connector/sources/{source}
• ingest.py > update `pipeline_spec` to reflect table(s) to test data ingestion with

This is basically how I have my setup configured. It *only* runs like this if I don't have `-e ` in my dependencies, otherwise, I get the errors I mentioned earlier.

Hope this helps!

[ShamenParis]

Hi @ChristianRRL ,

I found a couple of other options you can try:

1. Add the path directly to your pipeline JSON
Open your pipeline settings, switch to the JSON view, and add your workspace path under the libraries section. It should look like this:

"libraries": [
{
"files": {
"path": "/Workspace/Users/yourpath"
}
}
]

2. Package it as a Python Wheel (.whl)
Instead of pointing to raw Python files, you can package your custom code into a .whl file. Once you build it, just upload it to a Unity Catalog Volume (or your workspace) and attach it as a library in your pipeline settings.