- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-01-2025 07:57 PM
Hi kmodelew,
How are you doing today?, As per my understanding, this is a common confusion when using Databricks Asset Bundles (DAB) with multiple task groups and a shared codebase. The key thing to know is that DAB generates one .whl file for the entire bundle, which includes all your packages under src/, not separate wheel files for each task group. So when your YAML is looking for a wheel specific to task_group2, Databricks can’t find it because the wheel is named after your top-level project, not the individual packages.
To fix this, you just need to reference the correct package name (matching your top-level wheel) in the package_name: field of each job in your .yml files, and make sure your setup.py includes all sub-packages via find_packages() like you're already doing. So instead of trying to set package_name: task_group1 or task_group2, use the actual package name defined in setup.py (e.g. my_project), and in each job, point to the correct entry point function under that namespace (e.g. my_project.task_group1.main). That should fix the “wheel not found” error and let all task groups run off the same wheel file. Let me know if you want help adjusting the setup.py or yml—happy to take a look!
Regards,
Brahma
