Service Principal cannot access its own workspace folder

juan_barreto — Thu, 29 May 2025 21:07:15 GMT

We are using Asset bundles with databricks runtime 14.3LTS. During DAB deployment, the wheel is built and stored in the folder of the service principal running the deployment via GH workflow. The full path is
/Workspace/Users/SERVICE-PRINCIPAL-ID/.bundle/main_bundle/dev_dev/artifacts/.internal/WHEEL-NAME.whl

Within the same DAB, we define and deploy a job which uses a job cluster. when using the wheel as a task dependency we get this error message:

run failed with error message Library installation failed for library due to user error for whl: "/Workspace/Users/SERVICE-PRINCIPAL-ID/.bundle/main_bundle/dev_dev/artifacts/.internal/WHEEL-NAME.whl" Error messages: Library installation attempted on the driver node of cluster CLUSTER-ID and failed. User does not have permission to read the library file, or the file path does not exist. Error Code: FILE_NOT_FOUND_FAILURE. Error Message: java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File file:/Workspace/Users/SERVICE-PRINCIPAL-ID/.bundle/main_bundle/dev_dev/artifacts/.internalWHEEL-NAMEwhl does not exist

This was working fine a couple days before and now it is failing out of nowhere.

Re: Service Principal cannot access its own workspace folder

HariSankar — Fri, 30 May 2025 13:44:46 GMT

You're encountering a common issue when using service principals and job clusters with workspace-scoped paths. This typically happens due to
permission mismatches or cluster identity issues. Here’s a breakdown of the root cause and a recommended solution:

Root Cause:

When using a job cluster, the cluster runs under a different identity (not the same service principal that created the bundle).
If your `.whl` file is stored under:

/Workspace/Users/SERVICE-PRINCIPAL-ID/.bundle/...

the job cluster **may not have permission** to access that folder. This results in:

--> Library installation failed
--> java.io.FileNotFoundException` or `Permission Denied`

This behavior might have changed recently due to a permission policy update or workspace changes.

topic Service Principal cannot access its own workspace folder in Data Engineering

Service Principal cannot access its own workspace folder

Re: Service Principal cannot access its own workspace folder