topic Environment in serverless in Data Engineering

Environment in serverless

pepco — Fri, 31 Oct 2025 00:00:14 GMT

I'm playing little bit with on the Databricks free environment and I'm super confused by the documentation vs actual behavior. Maybe you could help me to understand better.

For the workspace I can define base environment which I can use in serverless compute. For example I defined mine as:

environment_version: '4' dependencies: - --index-url https://pypi.org/simple - databricks-sdk>=0.71.0 - databricks-labs-dqx>=0.9.3 - openpyxl

I started a notebook, changed the base environment and it worked. As per documentation:

"For jobs, only notebook tasks can use base environments".

I was really happy to see this because I played with serverless jobs before and I couldn't use environments in jobs. So I tried again with very simple job definition:

resources: jobs: New_Job_Oct_31_2025_12_37_AM: name: New Job Oct 31, 2025, 12:37 AM tasks: - task_key: test notebook_task: notebook_path: /Workspace/Users/hidden@gmail.com/test2 source: WORKSPACE environment_key: some_environment_key queue: enabled: true performance_target: PERFORMANCE_OPTIMIZED environments: - environment_key: some_environment_key spec: client: "4"

I know that the definition is not complete because I'm missing dependencies but upon saving the job definition, I'm still getting:

"A task environment can not be provided for notebook task test. Please use the %pip magic command to install notebook-scoped Python libraries and Python wheel packages".

Hence my confusion.

Is it possible to use environments with notebook tasks?

It's strange that it works with serverless in interactive notebook when I switch to my base environment, and doesn't work with job task.

Re: Environment in serverless

K_Anudeep — Fri, 31 Oct 2025 11:15:22 GMT

Hello @pepco ,

Is it possible to use environments with notebook tasks?

Yes—but only in a very specific way.

Notebook tasks can use base environments, but you don’t attach them in the job’s YAML. You pick the base env in the notebook’s Environment side panel, and notebook job runs inherit that. If you put environment_key on a notebook task, the Jobs API rejects it with the exact error you saw—that setting is for task environments on non-notebook tasks (Python script / wheel / dbt)

Doc:https://docs.databricks.com/aws/en/admin/workspace-settings/base-environment

Re: Environment in serverless

pepco — Fri, 31 Oct 2025 13:47:29 GMT

Hello,

You are correct I can choose environment in the notebook. But in my opinion documentation is miss-leading because it says: "For jobs, only notebook tasks can use base environments.".

Based on the explanation it basically means that it can be used only when notebook is run manually with attached serverless cluster using custom environment specification.