wheel package to install in a serveless workflow
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-13-2025 05:40 AM
Hi guys,
Which is the way through Databricks Asset Bundle to declare a new job definition having a serveless compute associated on each task that composes the workflow and be able that inside each notebook task definition is possible to catch the dependent custom libraries that I imported inside the workspace?
I did something like this:
environments:
- environment_key: envir
spec:
client: "1"
dependencies:
- "${workspace.root_path}/artifacts/.internal/data_pipelines-0.0.1-py3-none-any.whl"
tasks:
- task_key: schedule_next_run_for_this_job
description: due to business requirements is needed to reschedule the workflow in the near next run
environment_key: envir
notebook_task:
notebook_path: ../notebook/jobs/export.py
base_parameters:
function: schedule_next_run_for_this_job
env: ${bundle.target}
job_id: "{{job.id}}"
workspace_url: "{{workspace.url}}"but it returns to me:
Error: cannot create job: A task environment can not be provided for notebook task get_email_infos. Please use the %pip magic command to install notebook-scoped Python libraries and Python wheel packages
the only way to import personal wheel package inside a serveless compute is to install inside the notebook that library?
Because I want to do something like using:
libraries:
- whl: ...