topic Re: Set serveless compute environment to a task of a job in Data Engineering

Set serveless compute environment to a task of a job

jeremy98 — Wed, 29 Jan 2025 11:17:20 GMT

Hi Community,

I want to set the environment of a task inside in a job using DABs, but I got this error.

I could achieve my goal, if I set manually the task inside to be environment 2, because I need to use Python 3.11.

How can I do it through DABs?

Re: Set serveless compute environment to a task of a job

jeremy98 — Wed, 29 Jan 2025 11:43:31 GMT

Hi,

Seems that this could be set for spark_python_task:

resources: jobs: New_Job_Jan_29_2025_at_11_48_AM: name: New Job Jan 29, 2025 at 11:48 AM tasks: - task_key: test-py-version2 spark_python_task: python_file: /Workspace/Users/<private-email>/test-python-version2.py source: WORKSPACE environment_key: env_set queue: enabled: true environments: - environment_key: env_set spec: client: "2"

Why with a notebook task couldn't be set through DAB?

Re: Set serveless compute environment to a task of a job

saurabh18cs — Wed, 29 Jan 2025 13:49:33 GMT

for python notebooks you have to use notebook_task and for python code file you have to use

spark_python_task

Re: Set serveless compute environment to a task of a job

jeremy98 — Wed, 29 Jan 2025 13:53:25 GMT

Hi, yes I know but the point is different. I mean I would to set the environment version inside each task of a job. And seems that is possible to do it only using a spark_python_task instead of using notebook_task. And yes we are using py scripts as notebooks leveraging python magic from databricks.

Re: Set serveless compute environment to a task of a job

DelaneyClark — Fri, 31 Jan 2025 05:05:11 GMT

Thank you.

Re: Set serveless compute environment to a task of a job

saurabh18cs — Fri, 31 Jan 2025 12:08:48 GMT

To set environment variables in notebook_task, you can use the base_parameters field to pass parameters to the notebook. These parameters can then be accessed within the notebook using dbutils.widgets.

tasks:

- task_key: task1

notebook_task:

notebook_path: "/Users/your_user/notebook1"

base_parameters:

ENV_VERSION: "dev"