Databricks Terraform Provider Issues Passing Providers to Child Modules

absolutelyRice — Sat, 05 Nov 2022 17:19:27 GMT

I have been following the documentation on the terraform databricks documentation in order to provision account level resources on AWS. I can create the workspace fine, add users, etc... However, when I go to use the provider in non-mws mode, I am receiving errors saying:

 Error: workspace is most likely not created yet, because the `host` is empty. Please add `depends_on = [databricks_mws_workspaces.this]` or `depends_on = [azurerm_databricks_workspace.this]` to every data resource. See https://www.terraform.io/docs/language/resources/behavior.html more info. Please check https://registry.terraform.io/providers/databricks/databricks/latest/docs#authentication for details
│
│   with module.workspaces.data.databricks_spark_version.latest,
│   on ../modules/aws_workspaces/init.tf line 11, in data "databricks_spark_version" "latest":
│   11: data "databricks_spark_version" "latest" {}

To show how this created I have a file called root.tf which creates the root mws level resources absolutely fine.

provider "databricks" {
  alias    = "mws"
  host     = "https://accounts.cloud.databricks.com"
  username = var.databricks_account_username
  password = var.databricks_account_password
}
 
 
module "root" {
  source = "../modules/aws_root"
 
  databricks_account_id = var.databricks_account_id
 
  tags = var.tags
 
 
  region = var.region
 
  cidr_block = var.cidr_block
 
  databricks_users            = var.databricks_users
  databricks_metastore_admins = var.databricks_metastore_admins
 
  unity_admin_group = var.unity_admin_group
 
  providers = {
    databricks.mws = databricks.mws
  }
 
}

With outputs coming from that module:

output "databricks_host" {
  value = databricks_mws_workspaces.this.workspace_url
}
 
output "databricks_token" {
  value     = databricks_mws_workspaces.this.token[0].token_value
  sensitive = true
}
 
output "databricks_workspace_id" {
  value     = databricks_mws_workspaces.this.workspace_id
  sensitive = false
}
 
output "databricks_account_id" {
  value     = databricks_mws_workspaces.this.account_id
  sensitive = true
}
 
output "aws_iam_role_metastore_data_access_arn" {
  value = aws_iam_role.metastore_data_access.arn
}
 
output "aws_iam_role_metastore_data_access_name" {
  value = aws_iam_role.metastore_data_access.name
}
 
output "aws_s3_bucket_metastore_id" {
  value = aws_s3_bucket.metastore.id
}

These can be seen in the created resources when I do a

`terraform state show <outputs>`

However, when I go to create a workspace level provider to create some notebooks, clusters, etc... I seem to be unable to get the child module resources to use the newly created provider with a host, even though it is being set and I can see it's value. Even hard coding the host does not work. They all output the above error.

The creation of this provider and module can be seen here:

provider "databricks" {
  alias      = "workspace"
  host       = module.root.databricks_host
  token      = module.root.databricks_token
  account_id = module.root.databricks_account_id
}
 
 
 
 
module "workspaces" {
  source = "../modules/aws_workspaces"
 
  aws_s3_bucket_metastore_id              = module.root.aws_s3_bucket_metastore_id
  aws_iam_role_metastore_data_access_arn  = module.root.aws_iam_role_metastore_data_access_arn
  aws_iam_role_metastore_data_access_name = module.root.aws_iam_role_metastore_data_access_name
 
  cidr_block = var.cidr_block
 
  databricks_account_id        = var.databricks_account_id
  databricks_bronze_users      = var.databricks_bronze_users
  databricks_gold_users        = var.databricks_gold_users
  databricks_host              = module.root.databricks_host
  databricks_metastore_admins  = var.databricks_metastore_admins
  databricks_silver_users      = var.databricks_silver_users
  databricks_token             = module.root.databricks_token
  databricks_users             = var.databricks_users
  databricks_workspace_id      = module.root.databricks_workspace_id
  python_module_version_number = local.python_module_version_number
  shed_databricks_egg_name     = var.shed_databricks_egg_name
 
  tags              = var.tags
  unity_admin_group = var.unity_admin_group
  config            = local.config
 
  depends_on = [
    module.root
  ]
  providers = {
    databricks = databricks.workspace
  }
}

And the init method that uses this provider and is throwing the error can be seen here:

terraform {
  required_providers {
    databricks = {
      source                = "databricks/databricks"
      version               = "~> 1.6.2"
      configuration_aliases = [databricks.workspace]
    }
  }
}
 
data "databricks_spark_version" "latest" {}
data "databricks_node_type" "smallest" {
  local_disk = true
}

The suggestion to add a depends_on for the

`databricks_mws_workspace.this` isn't possible to do as it is created in the root module where the `databricks.mws` provider is used. (The documentation says each module should isolate providers.)

Re: Databricks Terraform Provider Issues Passing Providers to Child Modules

Jfoxyyc — Sat, 05 Nov 2022 17:26:07 GMT

I'm also curious about this as I haven't been able to successfully create an output for the databricks account_id.

Re: Databricks Terraform Provider Issues Passing Providers to Child Modules

absolutelyRice — Sun, 06 Nov 2022 08:41:33 GMT

That's odd, it makes me wonder if any attributes of that resource are getting exported, I suppose one could try to write the values out during an apply using a null resource and just echoing the values into a local file

Re: Databricks Terraform Provider Issues Passing Providers to Child Modules

absolutelyRice — Sun, 06 Nov 2022 17:11:32 GMT

So the answer to this was that you need to explicitly pass the provider argument to each of the data resources blocks. The docs should be updated to accommodate that.

i.e.

data "databricks_spark_version" "latest" {
  provider = databricks.workspace
}
data "databricks_node_type" "smallest" {
  provider = databricks.workspace
  local_disk = true
}

Re: Databricks Terraform Provider Issues Passing Providers to Child Modules

Jfoxyyc — Mon, 07 Nov 2022 22:37:09 GMT

Were you able to output the databricks account_id ?

Re: Databricks Terraform Provider Issues Passing Providers to Child Modules

absolutelyRice — Mon, 07 Nov 2022 22:41:21 GMT

I am going to be honest, I don't recall off the top of my head, but it is getting passed in as an argument to the other modules above so I assume so. I was able to verify that the other two were getting exported by adding a null resource that was something like:

    resource "null_resource" "echo" {
      local-exec = "echo '${module.root_mws.account_id}' > output.txt"
    }

topic Re: Databricks Terraform Provider Issues Passing Providers to Child Modules in Data Engineering

Databricks Terraform Provider Issues Passing Providers to Child Modules

Re: Databricks Terraform Provider Issues Passing Providers to Child Modules

Re: Databricks Terraform Provider Issues Passing Providers to Child Modules

Re: Databricks Terraform Provider Issues Passing Providers to Child Modules

Re: Databricks Terraform Provider Issues Passing Providers to Child Modules

Re: Databricks Terraform Provider Issues Passing Providers to Child Modules