cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to configure DAB bundles to run serverless

mlivshutz
New Contributor II

I am following the guidelines in https://docs.databricks.com/aws/en/dev-tools/bundles/jobs-tutorial to setup the job for serverless. It says to "omit the job_clusters configuration from the bundle configuration file." It sounds like the idea is to simply omit any mention of compute in the task configuration, and then Databricks will set the job for Serverless.

However, when I run bundle validate or bundle deploy, I get an error that I need to specify one of: job_cluster_key, environment_key, existing_cluster_id, new_cluster.

What do I need to do to enable serverless in the DAB configuration?

Note: this project is managed by Poetry, so that's why the DAB .yml file resides in the "resources" folder.

"Error: Missing required cluster or environment settings
at resources.jobs.dbx_backfill_emotion_job.tasks[0]
in resources/dbx_backfill_emotion_job.yml:38:11
databricks.yml:26:15

Task "dbx_backfill_emotion_main" requires a cluster or an environment to run.
Specify one of the following fields: job_cluster_key, environment_key, existing_cluster_id, new_cluster."

0 REPLIES 0

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group