Serverless base env setup in Databricks Asset Bundle (DAB)

[kranthit]

I am trying to set a base environment for my task (notebook) which is running on serverless, following is the dab yaml i am using when i did bundle deploy -t users, its not throwing any error but its not installing the libraries from the base env, can you please help me how to use base_environment in dabs

resources:
  jobs:
    SERVERLESS_JOB:
      name: ${var.env}_SERVERLESS_JOB_DAB
      parameters:
        - name: conf_catalog
          default: ${var.conf_catalog}
        - name: mode
          default: "full"
      tasks:
        - task_key: SERVERLESS_TASK
          notebook_task:
            notebook_path: ${var.src_file_path}/stack/test/serverless_task.py
          base_environment_path: "dbfs:/FileStore/Shared/scripts/serverless_base_env.yml"
          compute: "serverless"
      tags:
        project: "test"

 Thanks & Regards,

Kranthi

[Yogesh_378691]

Your YAML is valid, but the reason the libraries are not being installed is because base_environment_path is not supported for serverless compute. Serverless jobs use a fully managed environment and you can’t override it with a custom base environment.

If you need specific libraries, you have two options:

Use a non-serverless cluster and point base_environment_path to your YAML.

For serverless, install the required packages either by defining them under libraries in the job spec or by using %pip install inside the notebook.

In short, base_environment_path has no effect on serverless jobs.

[kranthit]

with the latest private preview we can provide base environment libraries from DAB yaml as follows, i am exploring if there is any way we can directly provide base_environment_path instad:

    environment_key: default
environments:
  - environment_key: default
    spec:
      environment_version: '3'
      dependencies:
        - --index-url https://pypi.org/simple
        - pgpy
        - paramiko
        - xxxx.whl