Hi @Daniele-T,

Thanks for sharing the exact error. It confirms this is not a quoting or syntax issue on your end.

This is more of a platform limitation, as far as I can tell. In the Jobs/DAB deployment path, base_environment is only accepted when it points to a custom environment spec file stored in Workspace files or a Unity Catalog Volume, for example, /Workspace/Shared/envs/ml-env.yml or /Volumes/my-catalog/my-schema/envs/ml-env.yml. Managed identifiers such as databricks_ml_v5 are rejected at the API layer regardless of the CLI version. I tested this in my own sandbox and reproduced the exact same error you hit, so this is consistent behaviour and not something specific to your setup. The serverless environment docs also note that if the base-environment preview is not enabled in a workspace, jobs expose environment_version rather than base_environment, and the Custom option expects a YAML file path.

So, for a scheduled Python script job deployed via Bundles and the CLI, you have two supported paths. The simpler one is to drop base_environment entirely and use environment_version with a small dependency list directly in your databricks.yml:

environments:
- environment_key: default
spec:
environment_version: "5"
dependencies:
- pandas==2.2.2
- scikit-learn==1.5.1

The second option, if you prefer to keep the environment spec separate from your bundle config, is to put the same spec in a YAML file, upload it to Workspace files or a UC Volume, and reference it via an absolute path:

environments:
- environment_key: default
spec:
base_environment: /Workspace/Users/your-user@company.com/envs/ml-env.yml

One important caveat on the second path. It is not equivalent to getting the Databricks ML runtime. The YAML file resolves to the same environment_version + dependencies mechanism, so you still need to list your packages explicitly. There is no way today to reference databricks_ml_v5 through the YAML path either.

On the startup time concern, it is worth knowing that serverless environments are cached, so after the first cold start, subsequent runs sharing the same dependency fingerprint will not reinstall. For a light dependency set, the overhead is smaller than it might seem. 

If your intention is specifically to get the full Databricks ML runtime pre-loaded (MLflow, Delta, the full ML stack) without listing packages, that is not supported in the DAB/CLI deployment path as of now. You would need to add those packages explicitly to your dependencies list for now.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.