Hi @jeremy98,
It appears that you are trying to use an environment block to specify dependencies for a notebook task, but this approach is not supported for notebook tasks on serverless compute. Instead, you should use the %pip magic command within the notebook to install the required libraries.
- Create the job definition with the necessary tasks. Each task should specify the notebook path and any parameters required
- Use the %pip magic command inside each notebook to install the custom libraries. This ensures that the libraries are available in the notebook's environment when the task runs.
Here’s an example:
bundle:
name: my-bundle
resources:
jobs:
my-job:
name: my-job
tasks:
- task_key: schedule_next_run_for_this_job
description: due to business requirements is needed to reschedule the workflow in the near next run
notebook_task:
notebook_path: /Workspace/Users/your_username/notebook/jobs/export.py
base_parameters:
function: schedule_next_run_for_this_job
env: ${bundle.target}
job_id: "{{job.id}}"
workspace_url: "{{workspace.url}}"
targets:
dev:
default: true
resources:
jobs:
my-job:
name: my-job
The example content of export.py:
# Install custom libraries using %pip magic command
%pip install /Workspace/Shared/Path/To/your_custom_library.whl
# Your notebook code here
def schedule_next_run_for_this_job():
# Function implementation
pass
# Call the function with parameters
schedule_next_run_for_this_job()
