topic Serverless base env setup in Databricks Asset Bundle (DAB) in Data Engineering

Serverless base env setup in Databricks Asset Bundle (DAB)

kranthit — Thu, 28 Aug 2025 16:25:32 GMT

I am trying to set a base environment for my task (notebook) which is running on serverless, following is the dab yaml i am using when i did bundle deploy -t users, its not throwing any error but its not installing the libraries from the base env, can you please help me how to use base_environment in dabs

resources: jobs: SERVERLESS_JOB: name: ${var.env}_SERVERLESS_JOB_DAB parameters: - name: conf_catalog default: ${var.conf_catalog} - name: mode default: "full" tasks: - task_key: SERVERLESS_TASK notebook_task: notebook_path: ${var.src_file_path}/stack/test/serverless_task.py base_environment_path: "dbfs:/FileStore/Shared/scripts/serverless_base_env.yml" compute: "serverless" tags: project: "test"

Thanks & Regards,

Kranthi

Re: Serverless base env setup in Databricks Asset Bundle (DAB)

Yogesh_Verma_ — Fri, 29 Aug 2025 03:38:20 GMT

Your YAML is valid, but the reason the libraries are not being installed is because base_environment_path is not supported for serverless compute. Serverless jobs use a fully managed environment and you can’t override it with a custom base environment.

If you need specific libraries, you have two options:

Use a non-serverless cluster and point base_environment_path to your YAML.

For serverless, install the required packages either by defining them under libraries in the job spec or by using %pip install inside the notebook.

In short, base_environment_path has no effect on serverless jobs.

Re: Serverless base env setup in Databricks Asset Bundle (DAB)

kranthit — Fri, 19 Sep 2025 15:00:56 GMT

with the latest private preview we can provide base environment libraries from DAB yaml as follows, i am exploring if there is any way we can directly provide base_environment_path instad:

    environment_key: default
environments:
  - environment_key: default
    spec:
      environment_version: '3'
      dependencies:
        - --index-url https://pypi.org/simple
        - pgpy
        - paramiko
        - xxxx.whl