We have implemented an asset bundle (DAB) that creates a wheel. During DAB deployment, the wheel is built and stored in the folder of the service principal running the deployment via GH workflow. The full path is
/Workspace/Users/SERVICE-PRINCIPAL-ID/.bundle/main_bundle/dev_dev/artifacts/.internal/WHEEL-NAME.whl
Within the same DAB, we define and deploy a job that is running on a shared general purpose cluster. The cluster is created via terraform and owned by the same service principal. Adding the wheel to tasks of this job via libraries-element results in a failure: LibraryInstallationError. Error message:
====SNIP=======
run failed with error message Library installation failed for library due to user error for whl: "/Workspace/Users/SERVICE-PRINCIPAL-ID/.bundle/main_bundle/dev_dev/artifacts/.internal/WHEEL-NAME.whl" Error messages: Library installation attempted on the driver node of cluster CLUSTER-ID and failed. User does not have permission to read the library file, or the file path does not exist. Error Code: FILE_NOT_FOUND_FAILURE. Error Message: java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File file:/Workspace/Users/SERVICE-PRINCIPAL-ID/.bundle/main_bundle/dev_dev/artifacts/.internalWHEEL-NAMEwhl does not exist
====SNAP=======
Listing the file path copied from that message using ls, dbutils, or even pip install works fine in a notebook. I can also verify that the file is owned {display as owned in the UI that is) by the service principal.
Any ideas?
