How to configure DAB bundles to run serverless
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Friday
I am following the guidelines in https://docs.databricks.com/aws/en/dev-tools/bundles/jobs-tutorial to setup the job for serverless. It says to "omit the job_clusters configuration from the bundle configuration file." It sounds like the idea is to simply omit any mention of compute in the task configuration, and then Databricks will set the job for Serverless.
However, when I run bundle validate or bundle deploy, I get an error that I need to specify one of: job_cluster_key, environment_key, existing_cluster_id, new_cluster.
What do I need to do to enable serverless in the DAB configuration?
Note: this project is managed by Poetry, so that's why the DAB .yml file resides in the "resources" folder.
"Error: Missing required cluster or environment settings
at resources.jobs.dbx_backfill_emotion_job.tasks[0]
in resources/dbx_backfill_emotion_job.yml:38:11
databricks.yml:26:15
Task "dbx_backfill_emotion_main" requires a cluster or an environment to run.
Specify one of the following fields: job_cluster_key, environment_key, existing_cluster_id, new_cluster."
- Labels:
-
Workflows
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Friday
Hey @mlivshutz ,
Right we just need to omit the job compute - can you make sure you are using latest databricks cli , if possible can you share an example of your databricks.yml file.
This might help - https://docs.databricks.com/aws/en/dev-tools/bundles/resource-examples#job-that-uses-serverless-comp...
Also make sure that your databricks cli is latest version
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
yesterday
Hi, @ashraf1395 ,
Thank you for looking at my question. My cli is 0.243, which is current as of today (3/17/25).
The task definition within resources/dbx_backfill_emotion_job.yml:
tasks:
- task_key: dbx_backfill_base_fields_x_1
# job_cluster_key: job_cluster
python_wheel_task:
package_name: dbx_backfill_emotion
entry_point: main
named_parameters: ¶meter_stub
source_path: x_smpl10_part_*
dbx_schemas_type: base_fields_schema
backfill_date_range: 2024-12-01 UPTO_BUT_EXCLUDING 2025-01-01
target_catalog: ${bundle.environment}_data_warehouse
target_schema: ${workspace.current_user.short_name}
target_table: x_base_fields_backfill_v2_dbxschema
libraries: &library_stub
# By default we just include the .whl file generated for the dbx_backfill_emotion package.
# See https://docs.databricks.com/dev-tools/bundles/library-dependencies.html
# for more information on how to add other libraries.
- whl: ../dist/*.whl
databricks.yml. I am deploying to dev target.
targets:
# The 'dev' target, for development purposes. This target is the default.
dev:
# We use 'mode: development' to indicate this is a personal development copy:
# - Deployed resources get prefixed with '[dev my_user_name]'
# - Any job schedules and triggers are paused by default
# - The 'development' mode is used for Delta Live Tables pipelines
# Your job runs as your service principal so any GCP/AWS permissions need to be assigned to you
mode: development
default: true
resources:
jobs:
dbx_backfill_emotion_job:
run_as:
user_name: ${workspace.current_user.userName}
# Override default settings here.
# tasks:
# - task_key: dbx_backfill_emotion_main
# python_wheel_task:
# named_parameters:
# target_schema: ${workspace.current_user.short_name}
workspace:
host: [redacted]

