Set serveless compute environment to a task of a job

[jeremy98]

Hi Community,

I want to set the environment of a task inside in a job using DABs, but I got this error.

I could achieve my goal, if I set manually the task inside to be environment 2, because I need to use Python 3.11.

How can I do it through DABs?

[jeremy98]

Hi,

Seems that this could be set for spark_python_task:

resources:
  jobs:
    New_Job_Jan_29_2025_at_11_48_AM:
      name: New Job Jan 29, 2025 at 11:48 AM
      tasks:
        - task_key: test-py-version2
          spark_python_task:
            python_file: /Workspace/Users/<private-email>/test-python-version2.py
            source: WORKSPACE
          environment_key: env_set
      queue:
        enabled: true
      environments:
        - environment_key: env_set
          spec:
            client: "2"

Why with a notebook task couldn't be set through DAB?

[saurabh18cs]

for python notebooks you have to use notebook_task and for python code file you have to use 

spark_python_task

[jeremy98]

Hi, yes I know but the point is different. I mean I would to set the environment version inside each task of a job. And seems that is possible to do it only using a spark_python_task instead of using notebook_task. And yes we are using py scripts as notebooks leveraging python magic from databricks.

[saurabh18cs]

To set environment variables in notebook_task, you can use the base_parameters field to pass parameters to the notebook. These parameters can then be accessed within the notebook using dbutils.widgets.

tasks:

      - task_key: task1

        notebook_task:

          notebook_path: "/Users/your_user/notebook1"

          base_parameters:

            ENV_VERSION: "dev"

[DelaneyClark]

Thank you.