Service principal seemingly cannot access its own workspace folder

Mathias_Peters — Wed, 10 Jul 2024 13:08:49 GMT

We have implemented an asset bundle (DAB) that creates a wheel. During DAB deployment, the wheel is built and stored in the folder of the service principal running the deployment via GH workflow. The full path is
/Workspace/Users/SERVICE-PRINCIPAL-ID/.bundle/main_bundle/dev_dev/artifacts/.internal/WHEEL-NAME.whl

Within the same DAB, we define and deploy a job that is running on a shared general purpose cluster. The cluster is created via terraform and owned by the same service principal. Adding the wheel to tasks of this job via libraries-element results in a failure: LibraryInstallationError. Error message:

====SNIP=======

run failed with error message Library installation failed for library due to user error for whl: "/Workspace/Users/SERVICE-PRINCIPAL-ID/.bundle/main_bundle/dev_dev/artifacts/.internal/WHEEL-NAME.whl" Error messages: Library installation attempted on the driver node of cluster CLUSTER-ID and failed. User does not have permission to read the library file, or the file path does not exist. Error Code: FILE_NOT_FOUND_FAILURE. Error Message: java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File file:/Workspace/Users/SERVICE-PRINCIPAL-ID/.bundle/main_bundle/dev_dev/artifacts/.internalWHEEL-NAMEwhl does not exist

====SNAP=======

Listing the file path copied from that message using ls, dbutils, or even pip install works fine in a notebook. I can also verify that the file is owned {display as owned in the UI that is) by the service principal.

Any ideas?

Re: Service principal seemingly cannot access its own workspace folder

Mathias_Peters — Wed, 10 Jul 2024 13:09:35 GMT

Update: When moving the whl to the shared workspace folder, the installation on the cluster works.

Re: Service principal seemingly cannot access its own workspace folder

RishabhTiwari07 — Thu, 18 Jul 2024 16:44:16 GMT

Thank you for sharing the solution that worked for you, I am sure it will help other community members.

Thanks
Rishabh

topic Re: Service principal seemingly cannot access its own workspace folder in Data Engineering

Service principal seemingly cannot access its own workspace folder

Re: Service principal seemingly cannot access its own workspace folder

Re: Service principal seemingly cannot access its own workspace folder