Databricks Community

BillBishop · ‎02-11-2025

using python_wheel_wrapper experimental true allows me to use python_wheel_task on an older cluster.

However, if I embed the python_wheel_task in a for_each_task it fails at runtime with:

"Library installation failed for library due to user error. Error message: Library from /Workspace is not allowed on this cluster. Please switch to using DBR 14.1+ ....."

It seems to me that there is a problem with for_each_task not honoring the experimental feature?

databricks.yml

experimental:
  python_wheel_wrapper: true

Task1 - works fine!

python_wheel_task:
  package_name: my_package
  entry_point: main
libraries:
  - whl: ../dist/*.whl

Task 2:

for_each_task:
  inputs: "{{tasks.<mystuff>}}"
  concurrency: ${var.ingest_concurrency}
  task:
    task_key: my_key
    existing_cluster_id: ${var.cluster_id}

    python_wheel_task:
      ...
    # Putting this inside the for_each_task fails on my cluster!
    libraries:
      - whl: ../dist/*.whl

Alberto_Umana · ‎02-11-2025

Hi @BillBishop,

I will check on this internally as outcome does not seem to be correct. If possible, upgrade your cluster to DBR 14.1 or later. This would resolve the issue without relying on the experimental feature