Hi @DineshOjha,

This is a good question, and researching this helped me learn some best practices along the way. What you’re seeing is actually expected behaviour. Service principals aren’t meant to execute notebooks directly from users’ personal workspace paths. That limitation is by design for security and isolation reasons.

Given you’re using Databricks Bundles and a notebooks‑first workflow, the recommended pattern is to treat Git as the source of truth. Developers can work on notebooks under their own /Workspace/Users/... paths (or locally) for convenience, then sync them to Git (via Git folders / Repos). Those copies in personal home directories should be considered development artefacts only, not what production jobs execute. In production, jobs should use notebooks deployed from Git into a shared workspace path, or reference Git directly (using jobs with a Git-based notebook source).

Instead of pointing jobs to /Workspace/Users/..., configure your bundle target so that it deploys notebooks into a shared folder where both the service principal has at least read/execute access, and your team can still inspect the deployed artefacts.

For example, in your bundle:

targets:
  prod:
    workspace:
      host: https://<your-workspace-url>
      root_path: /Workspace/Shared/projects/my-project

When you run databricks bundle deploy (ideally from CI/CD, authenticated as the service principal), the notebooks defined in the bundle are materialised under /Workspace/Shared/projects/my-project/... Your bundle’s jobs should reference those deployed notebook paths, not the originals under /Workspace/Users/....

On the Databricks side, you’ll typically want 

• On /Workspace/Shared/projects/my-project - Can Read (or higher) for the service principal, so it can read/execute the notebooks.
• On the jobs created by the bundle - Can Manage or Can Run for the service principal, depending on your governance model.
• On compute - Permission for the service principal to use the job cluster or shared compute resource configured for the bundle’s jobs.

With this setup developers continue to use their personal workspace areas for development. Git remains the source of truth. And, the service principal only interacts with the shared, deployed artifacts and never needs access to /Workspace/Users/....

If you prefer to be fully Git‑centric, you can also configure jobs to pull notebooks directly from Git (e.g. via Repos/git_source) and grant the service principal access to the Git repo, plus job permissions as above. However, the core principle is the same in both approaches... Don’t run production jobs against notebooks in /Workspace/Users/.... Use Git as the source of truth, and deploy or reference notebooks from a shared, service‑principal‑readable location.

Hope that helps clarify the pattern.

Please let me know if any of the above is unclear. 

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.