Environment in serverless

pepco
New Contributor III

‎10-30-2025 05:00 PM

I'm playing little bit with on the Databricks free environment and I'm super confused by the documentation vs actual behavior. Maybe you could help me to understand better.

For the workspace I can define base environment which I can use in serverless compute. For example I defined mine as:

environment_version: '4'
dependencies:
  - --index-url https://pypi.org/simple
  - databricks-sdk>=0.71.0
  - databricks-labs-dqx>=0.9.3
  - openpyxl

I started a notebook, changed the base environment and it worked. As per documentation:

"For jobs, only notebook tasks can use base environments".

I was really happy to see this because I played with serverless jobs before and I couldn't use environments in jobs. So I tried again with very simple job definition:

resources:
  jobs:
    New_Job_Oct_31_2025_12_37_AM:
      name: New Job Oct 31, 2025, 12:37 AM
      tasks:
        - task_key: test
          notebook_task:
            notebook_path: /Workspace/Users/hidden@gmail.com/test2
            source: WORKSPACE
          environment_key: some_environment_key
      queue:
        enabled: true
      performance_target: PERFORMANCE_OPTIMIZED
      environments:
        - environment_key: some_environment_key
          spec:
            client: "4"

I know that the definition is not complete because I'm missing dependencies but upon saving the job definition, I'm still getting:

"A task environment can not be provided for notebook task test. Please use the %pip magic command to install notebook-scoped Python libraries and Python wheel packages".

Hence my confusion.

Is it possible to use environments with notebook tasks?

It's strange that it works with serverless in interactive notebook  when I switch to my base environment, and doesn't work with job task.

 

0 Kudos

K_Anudeep
Databricks Employee
Databricks Employee

‎10-31-2025 04:15 AM

Hello @pepco ,

Is it possible to use environments with notebook tasks?

Yes—but only in a very specific way.

Notebook tasks can use base environments, but you don’t attach them in the job’s YAML. You pick the base env in the notebook’s Environment side panel, and notebook job runs inherit that. If you put environment_key on a notebook task, the Jobs API rejects it with the exact error you saw—that setting is for task environments on non-notebook tasks (Python script / wheel / dbt)

Doc:https://docs.databricks.com/aws/en/admin/workspace-settings/base-environment

Anudeep

View solution in original post

pepco
New Contributor III
In response to K_Anudeep

‎10-31-2025 06:47 AM

Hello, 

You are correct I can choose environment in the notebook. But in my opinion documentation is miss-leading because it says: "For jobs, only notebook tasks can use base environments.".

Based on the explanation it basically means that it can be used only when notebook is run manually with attached serverless cluster using custom environment specification.

0 Kudos