Problem:

When running dbt with persist column docs enabled we get the following error: org.apache.hadoop.hive.ql.metadata.HiveException: at least one column must be specified for the table

Background:

There is an issue on the dbt-spark github that was closed.

https://github.com/dbt-labs/dbt-spark/issues/364 and this was discussed in the #db-databricks-and-spark channel of the DBT slack community.

We tried the recommended fix, setting spark.databricks.delta.catalog.update.enabled=false, but we still get the error if column docs are enabled. We do not get the error if we set persist column docs to be false but then we are not able to use the full documentation functionality in dbt.

Questions:

1) a year ago there was a comment on the dbt slack "we're about 2 weeks away from deploying an enhancement to the async thread that does though delta catalog updates to fix this issue so you won't need to disable delta.catalog.update". Is the fix, above, still recommended? If not is there a new recommendation?

2) The recommended fix setting spark.databricks.delta.catalog.update.enabled=false worked for other posters but not for us so I am wondering if we are implementing correctly. The terraform docs do not exactly match what was described in the post. We set this parameter in "data_access_config" but posters describe setting it in "sql_config_params", but we are not able to do that with terraform. Is the implementation, below, correct? (links to docs that we looked at are bulleted, below)

• terraform docs on argument reference have sql_config_params separate from data_access_config as an optional map
• data access configuration supported properties includes spark.databricks.delta.catalog.update.enabled false
• sql endpoints global-edit documentation includes it in a separate sql_configuration_parameters in the Example Request (below)

resource "databricks_sql_global_config" "cfg" {

 instance_profile_arn = aws_iam_instance_profile.hc.arn

  data_access_config = {

  "spark.databricks.delta.catalog.update.enabled" : "false"

  }

}

resource "databricks_sql_global_config" "cfg" {

instance_profile_arn = aws_iam_instance_profile.hc.arn

sql_config_params = {

data_access_config = {

"spark.databricks.delta.catalog.update.enabled" : "false"

}

}

Thanks for your help!