Databricks asset bundles error "failed to instantiate provider"

[dbph]

Hi all,

I'm trying to deploy with databricks asset bundles. When running bundle deploy, the process fails with following error message:

failed execution pid=25092 exit_code=1 error="terraform apply: exit status 1\n\nError: failed to read schema for databricks_job.r005_job in registry.terraform.io/databricks/databricks: failed to instantiate provider \"registry.terraform.io/databricks/databricks\" to obtain schema: Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol.\n\n\nopen [...]\\terraform.tfstate: Das System kann die angegebene Datei nicht finden

CLI-Version: Databricks CLI v0.214.1

The ressource I'm trying to deploy is a simple example generated with bundle init:

# The main job for r005.
resources:
  jobs:
    r005_job:
      name: r005_job

      schedule:
        # Run every day at 8:37 AM
        quartz_cron_expression: '44 37 8 * * ?'
        timezone_id: Europe/Amsterdam

      tasks:
        - task_key: notebook_task
          job_cluster_key: job_cluster
          notebook_task:
            notebook_path: ../src/notebook.ipynb
        
      job_clusters:
        - job_cluster_key: job_cluster
          new_cluster:
            spark_version: 13.3.x-scala2.12
            node_type_id: Standard_D3_v2
            autoscale:
                min_workers: 1
                max_workers: 4

For my colleagues with the same setup, the deployment works perfectly. The deployment also works, when I deploy with a Azure DevOps deployment agent.

in the .databricks/bundle/dev... folder I can find a file "terraform-provider-databricks_v1.37.0.exe".

The bundle.tf.json file starts with:

{
  "terraform": {
    "required_providers": {
      "databricks": {
        "source": "databricks/databricks",
        "version": "1.37.0"
      }
    }
  },
  "provider": {
    "databricks": {}
  },
...

Does anyone have an idea, how I can get this to work?

Thanks!

[dbph]

Hi,

sorry for my late answer and thank you for your help. I was on parental leave in the meantime.

I solved the problem by installing the latest Databricks extension. With the new extension and updated Databricks-CLI I got the same error.

However: The latest Databricks extension has a "Refreh remote state" button in the bundle ressource explorer and voilà: Deployment works, no more errors 🙂

Thanks!