Asset Bundle API update issues

kanikeom — Tue, 18 Mar 2025 17:52:14 GMT

I was working on a proof of concept (POC) using the assert bundle. My job configuration in the .yml file worked yesterday, but it threw an error today during a demo to the team.

The error was likely due to an update to the Databricks API. After some trial and error and reviewing the API documentation, Adding the following to the job configuration resolved the issue:
kind: CLASSIC_PREVIEW

How can we anticipate such updates in the future and prevent these issues?

Here is the error message:
Error: terraform apply: exit status 1
Error: cannot update job: data_security_mode: DATA_SECURITY_MODE_DEDICATED is not allowed with unspecified kind.
with databricks_job.sample_python_job,
on bundle.tf.json line 69, in resource.databricks_job.sample_python_job:
69:

Re: Asset Bundle API update issues

Johannes_E — Wed, 23 Jul 2025 05:17:11 GMT

I've got the same issue and don't know how to fix it 😕

Re: Asset Bundle API update issues

mark_ott — Tue, 11 Nov 2025 01:33:07 GMT

Unexpected breaking changes to APIs—especially from cloud platforms like Databricks—can disrupt projects and demos. Proactively anticipating and rapidly adapting to such updates requires a combination of monitoring, process improvements, and technical safeguards.

Best Practices to Anticipate and Prevent API Breaking Changes

Monitor Official Sources

Subscribe to Databricks release notes and product update channels to learn about upcoming and recent changes.
Set up alerts from platforms such as GitHub repositories, provider changelogs (like Terraform Databricks provider), and Databricks community forums.

Pin Versions and Track Dependencies

Always pin specific versions of APIs, libraries, and providers (for example, the Databricks Terraform provider) in your configuration files. This helps ensure that your jobs use the known stable interface.
Regularly review your dependency version matrix and update intentionally, not automatically.

Test and Validate Regularly

Implement automated tests or pre-deployment checks using CI/CD pipelines to validate your configuration and code before every deployment.
Use staging environments that mirror production to test updates against the latest Databricks platform version.

Build Resilient Job Configurations

Specify all mandatory fields—such as kind: CLASSIC_PREVIEW—based on the latest API documentation so that your jobs do not rely on defaults that might change.
Track deprecation notices in API documentation and migrate off deprecated features early.

Engage with Databricks Support

Open support tickets or request clarifications when you encounter unexplained API behavior. Early feedback from support can guide your configuration changes.

Share Knowledge and Update Documentation

Document recent incidents, fixes (like adding kind: CLASSIC_PREVIEW), and lessons learned on internal wikis or playbooks.
Provide regular internal updates so your team is aware of how to resolve similar issues.

Error Cause Analysis

The error message you encountered:

Error: cannot update job: data_security_mode: DATA_SECURITY_MODE_DEDICATED is not allowed with unspecified kind.

shows Databricks changed the enforcement around the kind parameter when certain security modes are set. “Unspecified kind” is now invalid with some security settings; explicitly stating kind: CLASSIC_PREVIEW resolved the incompatibility.

Regularly checking for these sorts of changes to parameter requirements in the API documentation is essential to avoid future breakages.

Resources for Staying Up to Date

Databricks Release Notes and Changelog: https://docs.databricks.com/release-notes/index.html
Databricks Terraform Provider Changelog: https://registry.terraform.io/providers/databricks/databricks/latest/docs/guides/provider-versioning
Databricks Community Forum: https://community.databricks.com/

By combining active monitoring, disciplined dependency management, automated validation, and internal communication, your team can reduce the risk and impact of API updates causing POC or production failures.

topic Re: Asset Bundle API update issues in Data Engineering

Asset Bundle API update issues

Re: Asset Bundle API update issues

Re: Asset Bundle API update issues

Best Practices to Anticipate and Prevent API Breaking Changes

Monitor Official Sources

Pin Versions and Track Dependencies

Test and Validate Regularly

Build Resilient Job Configurations

Engage with Databricks Support

Share Knowledge and Update Documentation

Error Cause Analysis

Resources for Staying Up to Date