topic ModuleNotFoundError or Permission denied are occurring when running Job with Git and All purpose in Data Engineering

ModuleNotFoundError or Permission denied are occurring when running Job with Git and All purpose

Yuki — Tue, 27 May 2025 14:08:26 GMT

"ModuleNotFoundError" or "Permission denied" are occurring when running Jobs with Git provider and All purpose clusters.

Which one occurs depends on the workspace, but it is reproducible.

Is that no longer recommended?

I can run it by serverless cluster.
I can get the notebook from github, but I can't get modules well.
I can run it when I'm using workspace notebooks.
Cluster Details
- AWS
- 15.4 LTS
- Dedicated access mode / Using Group
Job Details
- "Run as" is using personal account

Re: ModuleNotFoundError or Permission denied are occurring when running Job with Git and All purpose

Renu_ — Mon, 02 Jun 2025 13:52:38 GMT

Hi @Yuki, these errors often stem from how Git provider integration behaves with All-Purpose clusters in Dedicated access mode. "Permission denied" usually means the Git credentials (like a personal access token) aren’t properly inherited by the job, unlike Serverless clusters, which handle this more smoothly.

"ModuleNotFoundError" happens when the cluster doesn't recognize the module path from the Git folder. This often requires manually updating sys.path or using absolute imports. To resolve:

Ensure your Git credentials are correctly set with proper scope.
Add the repo root to sys.path in your notebook.

Re: ModuleNotFoundError or Permission denied are occurring when running Job with Git and All purpose

Yuki — Tue, 03 Jun 2025 03:08:35 GMT

Hi @Renu_ ,
Thank you so much for your advice. I will try to both patterns.
Regards,