Hi @Daniele-T,

ML is the UI label for a Databricks-managed serverless base environment, but in Jobs/DAB you generally need to use the job environment model rather than the notebook picker directly. Databricks documents that the serverless base environment options include Standard, ML, AI, previous versions, Custom (YAML), and workspace environments, and that job tasks are configured through the job environment settings.

If this is a notebook task, the simplest option may be to let the task use the notebook's own environment, because notebook tasks default to Notebook Environment unless you override them with a job-level environment. See Configure the serverless environment.

If you do want to define the environment in DAB, base_environment: "ML" is not the right value. For managed base environments, Databricks uses versioned identifiers, and Databricks-provided ML environments are versioned like databricks_ml_v5. The public environment APIs also describe Databricks-provided ML base environments as workspace-base-environments/databricks_ml_..., for example workspace-base-environments/databricks_ml_v5.

So the configuration should look more like this:

environments:
  - environment_key: default
    spec:
      base_environment: databricks_ml_v5
      dependencies:
        - ...

In that case, do not also set environment_version in the same spec.

A second thing to check is workspace support. Databricks notes that selecting a managed base environment for jobs is in beta, and that if the workspace does not have that feature enabled, the job configuration shows an Environment version drop-down instead of Base environment. In those workspaces, the "Custom" option expects a YAML file, which matches what you are seeing. See Configure the serverless environment.

So... 

• If this is a notebook task, consider configuring the notebook itself to use the ML base environment and let the job use Notebook Environment by default.
• If you want a job-level environment and your workspace supports managed base environments for jobs, use base_environment: databricks_ml_v5 plus your light dependencies.
• If your workspace does not support managed base environments for jobs yet, then the supported routes are either environment_version plus dependencies, or a Custom YAML-based environment.

Hope this helps.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.