Databricks Bundle Validation Error After CLI Upgrade (0.274.0 → 0.276.0)

databricksero
New Contributor II

‎11-06-2025 12:50 AM

After upgrading the Databricks CLI from version 0.274.0 to 0.276.0, bundle validation is failing with an error indicating that my configuration is formatted for "open-source Spark Declarative Pipelines" while the CLI now only supports "Lakeflow Declarative Pipelines".

Error MessageError: /path/to/databricks-asset-bundle/jobs/jobs.yml seems to be formatted for open-source Spark Declarative Pipelines. Pipelines CLI currently only supports Lakeflow Declarative Pipelines development. To see an example of a supported pipelines template, create a new Pipelines CLI project with "pipelines init". 

This configuration was working correctly with CLI version 0.274.0. The error occurs when running:databricks bundle validate --target dev 

Configuration Files

pipelines.yml

# reusable yaml anchors
definitions:
environment: &environment
dependencies:
- ../../dist/data_platform*.whl

permissions: &permissions
- group_name: users
level: CAN_RUN


resources:
pipelines:
example_pipeline:
name: example-pipeline
serverless: true
channel: PREVIEW
catalog: ${var.catalog}
schema: ${resources.schemas.bronze_schema.name}
root_path: ${workspace.root_path}/files
environment: *environment
permissions: *permissions
configuration:
BRONZE_SCHEMA: ${var.catalog}.${resources.schemas.bronze_schema.name}
SILVER_SCHEMA: ${var.catalog}.${resources.schemas.silver_schema.name}
libraries:
- file:
path: ${var.datasources_path}/datasource/example/bronze_pipeline.py
- file:
path: ${var.datasources_path}/datasource/example/silver_pipeline.py

jobs.yml

# reusable yaml anchors
definitions:
environments: &environments
- environment_key: default
spec:
environment_version: "3"
dependencies:
- ../../dist/data_platform*.whl

permissions: &permissions
- group_name: users
level: CAN_MANAGE_RUN

resources:
jobs:
# Example data source job
example_job:
name: example-full-job
tasks:
- task_key: example_ingestion
spark_python_task:
python_file: ${var.datasources_path}/datasource/example/ingest.py
source: WORKSPACE
environment_key: default
- task_key: example_pipeline
pipeline_task:
pipeline_id: ${resources.pipelines.example_pipeline.id}
depends_on:
- task_key: example_ingestion
schedule:
quartz_cron_expression: "0 0 6 ? * MON" # Weekly on Monday at 6 AM UTC (7 AM CET)
timezone_id: "UTC"
pause_status: "UNPAUSED"
email_notifications: ${var.email_notifications}
permissions: *permissions
environments: *environments

 

Questions

 

1. What changed between CLI versions 0.274.0 and 0.276.0 that causes this validation error?
2. Is there a migration path or configuration change required to make existing bundle configurations compatible with the new CLI version?
3. Are there any breaking changes documented for this upgrade that I should be aware of?

 

Any guidance on resolving this issue would be greatly appreciated.

 

0 Kudos