using python_wheel_wrapper experimental true allows me to use python_wheel_task on an older cluster.
However, if I embed the python_wheel_task in a for_each_task it fails at runtime with:
"Library installation failed for library due to user error. Error message: Library from /Workspace is not allowed on this cluster. Please switch to using DBR 14.1+ ....."
It seems to me that there is a problem with for_each_task not honoring the experimental feature?
databricks.yml
experimental:
python_wheel_wrapper: true
Task1 - works fine!
python_wheel_task:
package_name: my_package
entry_point: main
libraries:
- whl: ../dist/*.whl
Task 2:
for_each_task:
inputs: "{{tasks.<mystuff>}}"
concurrency: ${var.ingest_concurrency}
task:
task_key: my_key
existing_cluster_id: ${var.cluster_id}
python_wheel_task:
...
# Putting this inside the for_each_task fails on my cluster!
libraries:
- whl: ../dist/*.whl