You're encountering a common issue when using service principals and job clusters with workspace-scoped paths. This typically happens due to
permission mismatches or cluster identity issues. Here’s a breakdown of the root cause and a recommended solution:
Root Cause:
When using a job cluster, the cluster runs under a different identity (not the same service principal that created the bundle).
If your `.whl` file is stored under:
/Workspace/Users/SERVICE-PRINCIPAL-ID/.bundle/...
the job cluster **may not have permission** to access that folder. This results in:
--> Library installation failed
--> java.io.FileNotFoundException` or `Permission Denied`
This behavior might have changed recently due to a permission policy update or workspace changes.
Recommended Solution:
1.Avoid storing `.whl` under the service principal's folder
Instead, store the `.whl` in a shared location accessible by all clusters. Use **DBFS** or **Workspace shared folders**, such as:
dbfs:/FileStore/libs/your_wheel.whl
Then update your job/library reference like:
"libraries": [
{
"whl": "dbfs:/FileStore/libs/your_wheel.whl"
}
]
2.Check and adjust permissions (if still using workspace path):
If you must use `/Workspace/...` paths:
--> Navigate to the file or folder in the Workspace UI.
--> Click “Permissions” and ensure the service principal or cluster owner has **Can Read** or **Can Manage** permissions.
3.Alternative Approach – Use a repo or MLflow artifacts
Consider storing your wheel file in:
--> A Git repo connected to Databricks Repos
--> MLflow registered models or artifacts
--> A cloud storage location (S3, ADLS, GCS) with appropriate IAM access
harisankar
