Databricks asset bundles error "failed to instantiate provider"

dbph — Fri, 23 Feb 2024 17:16:00 GMT

Hi all,

I'm trying to deploy with databricks asset bundles. When running bundle deploy, the process fails with following error message:

failed execution pid=25092 exit_code=1 error="terraform apply: exit status 1\n\nError: failed to read schema for databricks_job.r005_job in registry.terraform.io/databricks/databricks: failed to instantiate provider \"registry.terraform.io/databricks/databricks\" to obtain schema: Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol.\n\n\nopen [...]\\terraform.tfstate: Das System kann die angegebene Datei nicht finden

CLI-Version: Databricks CLI v0.214.1

The ressource I'm trying to deploy is a simple example generated with bundle init:

# The main job for r005. resources: jobs: r005_job: name: r005_job schedule: # Run every day at 8:37 AM quartz_cron_expression: '44 37 8 * * ?' timezone_id: Europe/Amsterdam tasks: - task_key: notebook_task job_cluster_key: job_cluster notebook_task: notebook_path: ../src/notebook.ipynb job_clusters: - job_cluster_key: job_cluster new_cluster: spark_version: 13.3.x-scala2.12 node_type_id: Standard_D3_v2 autoscale: min_workers: 1 max_workers: 4

For my colleagues with the same setup, the deployment works perfectly. The deployment also works, when I deploy with a Azure DevOps deployment agent.

in the .databricks/bundle/dev... folder I can find a file "terraform-provider-databricks_v1.37.0.exe".

The bundle.tf.json file starts with:

{ "terraform": { "required_providers": { "databricks": { "source": "databricks/databricks", "version": "1.37.0" } } }, "provider": { "databricks": {} }, ...

Does anyone have an idea, how I can get this to work?

Thanks!

Re: Databricks asset bundles error "failed to instantiate provider"

dbph — Sun, 23 Jun 2024 20:22:55 GMT

Hi,

sorry for my late answer and thank you for your help. I was on parental leave in the meantime.

I solved the problem by installing the latest Databricks extension. With the new extension and updated Databricks-CLI I got the same error.

However: The latest Databricks extension has a "Refreh remote state" button in the bundle ressource explorer and voilà: Deployment works, no more errors 🙂

Thanks!

topic Databricks asset bundles error "failed to instantiate provider" in Data Engineering

Databricks asset bundles error "failed to instantiate provider"

Re: Databricks asset bundles error "failed to instantiate provider"