ShamenParis
Contributor

Hi @ChristianRRL ,
Your compute cluster is caching the older version of your Python package in memory. Standard job runs or dry runs don't always force a rebuild or reload of workspace files.

I hope below steps will helps you,

  1. Trigger a Full Refresh: Instead of a regular run or dry run, trigger a Full Refresh on the pipeline. This forces the engine to re-evaluate the code and pull the latest workspace files.

  2. Use Editable Install: Ensure your pipeline dependencies are installing your custom package in editable mode by adding -e (e.g., pip install -e /Workspace/Users/...). This links directly to the source files rather than a static build.

  3. Develop Interactively (Best for Live Testing): Testing via Job Compute is too slow for active development. Attach a standard Notebook to an All-Purpose cluster, run %load_ext autoreload followed by %autoreload 2, and test your connector logic there first. It will instantly pick up any changes you make to {source}_schemas.py without needing to rebuild.