DAB for_each_task python wheel fail

BillBishop — Tue, 11 Feb 2025 17:40:57 GMT

using python_wheel_wrapper experimental true allows me to use python_wheel_task on an older cluster.

However, if I embed the python_wheel_task in a for_each_task it fails at runtime with:

"Library installation failed for library due to user error. Error message: Library from /Workspace is not allowed on this cluster. Please switch to using DBR 14.1+ ....."

It seems to me that there is a problem with for_each_task not honoring the experimental feature?

databricks.yml

experimental:
  python_wheel_wrapper: true

Task1 - works fine!

python_wheel_task:
  package_name: my_package
  entry_point: main
libraries:
  - whl: ../dist/*.whl

Task 2:

for_each_task:
  inputs: "{{tasks.<mystuff>}}"
  concurrency: ${var.ingest_concurrency}
  task:
    task_key: my_key
    existing_cluster_id: ${var.cluster_id}

    python_wheel_task:
      ...
    # Putting this inside the for_each_task fails on my cluster!
    libraries:
      - whl: ../dist/*.whl

Re: DAB for_each_task python wheel fail

Alberto_Umana — Tue, 11 Feb 2025 20:28:59 GMT

Hi @BillBishop,

I will check on this internally as outcome does not seem to be correct. If possible, upgrade your cluster to DBR 14.1 or later. This would resolve the issue without relying on the experimental feature

Re: DAB for_each_task python wheel fail

BillBishop — Wed, 19 Feb 2025 14:38:28 GMT

it works as expected in later runtime versions. Thanks, we run with upgraded DBR.

topic Re: DAB for_each_task python wheel fail in Data Engineering

DAB for_each_task python wheel fail

Re: DAB for_each_task python wheel fail

Re: DAB for_each_task python wheel fail