I am following the guidelines in https://docs.databricks.com/aws/en/dev-tools/bundles/jobs-tutorial to setup the job for serverless. It says to "omit the job_clusters configuration from the bundle configuration file." It sounds like the idea is to simply omit any mention of compute in the task configuration, and then Databricks will set the job for Serverless.
However, when I run bundle validate or bundle deploy, I get an error that I need to specify one of: job_cluster_key, environment_key, existing_cluster_id, new_cluster.
What do I need to do to enable serverless in the DAB configuration?
Note: this project is managed by Poetry, so that's why the DAB .yml file resides in the "resources" folder.
"Error: Missing required cluster or environment settings
at resources.jobs.dbx_backfill_emotion_job.tasks[0]
in resources/dbx_backfill_emotion_job.yml:38:11
databricks.yml:26:15
Task "dbx_backfill_emotion_main" requires a cluster or an environment to run.
Specify one of the following fields: job_cluster_key, environment_key, existing_cluster_id, new_cluster."
