Problem:
When running dbt with persist column docs enabled we get the following error: org.apache.hadoop.hive.ql.metadata.HiveException: at least one column must be specified for the table
Background:
There is an issue on the dbt-spark github that was closed.
https://github.com/dbt-labs/dbt-spark/issues/364 and this was discussed in the #db-databricks-and-spark channel of the DBT slack community.
We tried the recommended fix, setting spark.databricks.delta.catalog.update.enabled=false, but we still get the error if column docs are enabled. We do not get the error if we set persist column docs to be false but then we are not able to use the full documentation functionality in dbt.
Questions:
1) a year ago there was a comment on the dbt slack "we're about 2 weeks away from deploying an enhancement to the async thread that does though delta catalog updates to fix this issue so you won't need to disable delta.catalog.update". Is the fix, above, still recommended? If not is there a new recommendation?
2) The recommended fix setting spark.databricks.delta.catalog.update.enabled=false worked for other posters but not for us so I am wondering if we are implementing correctly. The terraform docs do not exactly match what was described in the post. We set this parameter in "data_access_config" but posters describe setting it in "sql_config_params", but we are not able to do that with terraform. Is the implementation, below, correct? (links to docs that we looked at are bulleted, below)
resource "databricks_sql_global_config" "cfg" {
instance_profile_arn = aws_iam_instance_profile.hc.arn
data_access_config = {
"spark.databricks.delta.catalog.update.enabled" : "false"
}
}
resource "databricks_sql_global_config" "cfg" {
instance_profile_arn = aws_iam_instance_profile.hc.arn
sql_config_params = {
data_access_config = {
"spark.databricks.delta.catalog.update.enabled" : "false"
}
}
Thanks for your help!
