Databricks asset bundles error "failed to instantiate provider"

[dbph]

Hi all,

I'm trying to deploy with databricks asset bundles. When running bundle deploy, the process fails with following error message:

failed execution pid=25092 exit_code=1 error="terraform apply: exit status 1\n\nError: failed to read schema for databricks_job.r005_job in registry.terraform.io/databricks/databricks: failed to instantiate provider \"registry.terraform.io/databricks/databricks\" to obtain schema: Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol.\n\n\nopen [...]\\terraform.tfstate: Das System kann die angegebene Datei nicht finden

CLI-Version: Databricks CLI v0.214.1

The ressource I'm trying to deploy is a simple example generated with bundle init:

# The main job for r005.
resources:
  jobs:
    r005_job:
      name: r005_job

      schedule:
        # Run every day at 8:37 AM
        quartz_cron_expression: '44 37 8 * * ?'
        timezone_id: Europe/Amsterdam

      tasks:
        - task_key: notebook_task
          job_cluster_key: job_cluster
          notebook_task:
            notebook_path: ../src/notebook.ipynb
        
      job_clusters:
        - job_cluster_key: job_cluster
          new_cluster:
            spark_version: 13.3.x-scala2.12
            node_type_id: Standard_D3_v2
            autoscale:
                min_workers: 1
                max_workers: 4

For my colleagues with the same setup, the deployment works perfectly. The deployment also works, when I deploy with a Azure DevOps deployment agent.

in the .databricks/bundle/dev... folder I can find a file "terraform-provider-databricks_v1.37.0.exe".

The bundle.tf.json file starts with:

{
  "terraform": {
    "required_providers": {
      "databricks": {
        "source": "databricks/databricks",
        "version": "1.37.0"
      }
    }
  },
  "provider": {
    "databricks": {}
  },
...

Does anyone have an idea, how I can get this to work?

Thanks!

[Kaniz]

Hi @dbph , It seems you’re encountering an issue with deploying Databricks Asset Bundles.

Let’s troubleshoot this step by step.

1. Terraform Provider Issue: The error message indicates a problem with the Terraform provider for Databricks. Specifically, it mentions that the plugin failed to read the schema for databricks_job.r005_job. Here are some steps to address this:

   • Plugin Recompilation: The error suggests that the plugin might need recompilation to support the latest protocol. Ensure that you have the correct version of the Terraform provider for Databricks (version 1.37.0) installed. If not, try updating or reinstalling it.

   • Check Plugin Validity: Verify that the terraform-provider-databricks_v1.37.0.exe file in your .databricks/bundle/dev... folder is valid and not corrupted. If necessary, download it again from a reliable source.

2. Bundle Configuration: Let’s review your bundle.tf.json file:

   {
     "terraform": {
       "required_providers": {
         "databricks": {
           "source": "databricks/databricks",
           "version": "1.37.0"
         }
       }
     },
     "provider": {
       "databricks": {}
     }
     // ... (other configurations)
   }
   

   • Ensure that the databricks provider is correctly specified with the desired version (1.37.0). Double-check the syntax and indentation.

3. Deployment Modes: Databricks Asset Bundles support different deployment modes: development and production. Make sure you’re deploying in the intended mode. Here’s how to specify the mode:

   • Development Mode: To deploy in development mode, add the following mode mapping to your target configuration:

     targets:
       dev:
         mode: development
     

     Deployment in development mode includes behaviors like tagging jobs as “dev,” using compute IDs, and pausing schedules.

   • Production Mode: For production deployments, add the following mode mapping:

     targets:
       prod:
         mode: production
     

     In production mode, validations related to Git branches and Delta Live Tables pipelines occur.

4. Environment Variables: Ensure that your environment variables (such as DATABRICKS_HOST and DATABRICKS_TOKEN) are correctly set. These variables are crucial for connecting to your Databricks workspace.

5. Git Branch Validation: If you’re using Git, verify that the current Git branch matches the one specified in your target configuration.

   You can set the Git branch in the target like this:

   git:
     branch: main
   

   Use the --force flag during deployment to override Git branch validation if needed.

6. Debugging: If the issue persists, consider checking the paths in your .databricks/bundle/my_project_job.yml file. Ensure they point to the correct locations.

Remember that Databricks Asset Bundles simplify CI/CD workflows, so let’s work through these steps to get your deployment working smoothly! 🚀