cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

ModuleNotFoundError or Permission denied are occurring when running Job with Git and All purpose

Yuki
Contributor

"ModuleNotFoundError" or "Permission denied" are occurring when running Jobs with Git provider and All purpose clusters.

Which one occurs depends on the workspace, but it is reproducible.

Is that no longer recommended?

Yuki_1-1748354429378.png

Yuki_2-1748354836276.png

  • I can run it by serverless cluster.
  • I can get the notebook from github, but I can't get modules well.
  • I can run it when I'm using workspace notebooks.
  • Cluster Details
    • AWS
    • 15.4 LTS
    • Dedicated access mode / Using Group
  • Job Details
    • "Run as" is using personal account
2 REPLIES 2

Renu_
Contributor III

Hi @Yuki, these errors often stem from how Git provider integration behaves with All-Purpose clusters in Dedicated access mode. "Permission denied" usually means the Git credentials (like a personal access token) aren’t properly inherited by the job, unlike Serverless clusters, which handle this more smoothly.

"ModuleNotFoundError" happens when the cluster doesn't recognize the module path from the Git folder. This often requires manually updating sys.path or using absolute imports. To resolve:

  • Ensure your Git credentials are correctly set with proper scope.
  • Add the repo root to sys.path in your notebook.

Yuki
Contributor

Hi @Renu_ ,
Thank you so much for your advice. I will try to both patterns.
Regards,

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now