cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

DAB for_each_task python wheel fail

BillBishop
New Contributor III

using python_wheel_wrapper experimental true allows me to use python_wheel_task on an older cluster.

However, if I embed the python_wheel_task in a for_each_task it fails at runtime with:

 "Library installation failed for library due to user error.  Error message: Library from /Workspace is not allowed on this cluster.  Please switch to using DBR 14.1+ ....."

It seems to me that there is a problem with for_each_task not honoring the experimental feature?

databricks.yml

experimental:
python_wheel_wrapper: true

Task1 - works fine!

python_wheel_task:
package_name: my_package
entry_point: main
libraries:
- whl: ../dist/*.whl

 

Task 2:

 

for_each_task:
inputs: "{{tasks.<mystuff>}}"
concurrency: ${var.ingest_concurrency}
task:
task_key: my_key
existing_cluster_id: ${var.cluster_id}

python_wheel_task:
...
# Putting this inside the for_each_task fails on my cluster!
libraries:
- whl: ../dist/*.whl

 

 

 

 

2 REPLIES 2

Alberto_Umana
Databricks Employee
Databricks Employee

Hi @BillBishop,

I will check on this internally as outcome does not seem to be correct. If possible, upgrade your cluster to DBR 14.1 or later. This would resolve the issue without relying on the experimental feature

it works as expected in later runtime versions.  Thanks, we run with upgraded DBR.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group