cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Serverless base env setup in Databricks Asset Bundle (DAB)

kranthit
New Contributor

I am trying to set a base environment for my task (notebook) which is running on serverless, following is the dab yaml i am using when i did bundle deploy -t users, its not throwing any error but its not installing the libraries from the base env, can you please help me how to use base_environment in dabs

resources:
  jobs:
    SERVERLESS_JOB:
      name: ${var.env}_SERVERLESS_JOB_DAB
      parameters:
        - name: conf_catalog
          default: ${var.conf_catalog}
        - name: mode
          default: "full"
      tasks:
        - task_key: SERVERLESS_TASK
          notebook_task:
            notebook_path: ${var.src_file_path}/stack/test/serverless_task.py
          base_environment_path: "dbfs:/FileStore/Shared/scripts/serverless_base_env.yml"
          compute: "serverless"
      tags:
        project: "test"

 Thanks & Regards,

Kranthi

1 REPLY 1

Yogesh_378691
Contributor

Your YAML is valid, but the reason the libraries are not being installed is because base_environment_path is not supported for serverless compute. Serverless jobs use a fully managed environment and you canโ€™t override it with a custom base environment.

If you need specific libraries, you have two options:

Use a non-serverless cluster and point base_environment_path to your YAML.

For serverless, install the required packages either by defining them under libraries in the job spec or by using %pip install inside the notebook.

In short, base_environment_path has no effect on serverless jobs.

Yogesh Verma

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local communityโ€”sign up today to get started!

Sign Up Now