May I know the exact error you are getting on the cluster?
You can use the following code to use a wheel in a shared folder:
resources:
jobs:
my-job:
name: my-job
tasks:
- task_key: my-task
new_cluster:
spark_version: 13.3.x-scala2.12
node_type_id: i3.xlarge
data_security_mode: USER_ISOLATION
num_workers: 1
python_wheel_task:
entry_point: main
package_name: my_package
libraries:
- whl: /Shared/path/to/your/wheel-file.whl