You're encountering a common issue when using service principals and job clusters with workspace-scoped paths. This typically happens due to
permission mismatches or cluster identity issues. Here’s a breakdown of the root cause and a recommended solution:

Root Cause:

When using a job cluster, the cluster runs under a different identity (not the same service principal that created the bundle).
If your `.whl` file is stored under:

/Workspace/Users/SERVICE-PRINCIPAL-ID/.bundle/...

the job cluster **may not have permission** to access that folder. This results in:

--> Library installation failed
--> java.io.FileNotFoundException` or `Permission Denied`

This behavior might have changed recently due to a permission policy update or workspace changes.

Recommended Solution:

1.Avoid storing `.whl` under the service principal's folder

Instead, store the `.whl` in a shared location accessible by all clusters. Use **DBFS** or **Workspace shared folders**, such as:

dbfs:/FileStore/libs/your_wheel.whl

Then update your job/library reference like:

"libraries": [
{
"whl": "dbfs:/FileStore/libs/your_wheel.whl"
}
]

2.Check and adjust permissions (if still using workspace path):

If you must use `/Workspace/...` paths:

--> Navigate to the file or folder in the Workspace UI.
--> Click “Permissions” and ensure the service principal or cluster owner has **Can Read** or **Can Manage** permissions.

3.Alternative Approach – Use a repo or MLflow artifacts

Consider storing your wheel file in:

--> A Git repo connected to Databricks Repos
--> MLflow registered models or artifacts
--> A cloud storage location (S3, ADLS, GCS) with appropriate IAM access