topic Re: Best way to automatically update a delta table schema in Data Engineering

Best way to automatically update a delta table schema

hari — Fri, 14 Oct 2022 13:16:00 GMT

We have multiple environments where the same tables are added so it's really hard to manually update the schema of the table across all the environments. We know that it's not ideal to update table schema a lot but our product is still evolving and schema changes will occur more frequently than in an ideal case.

I see that databricks has released a databricks_table resource into the public preview. Really excited to see how it evolves.

Does anyone know if terraform stack apply can handle changes to the schema?

If so, How does it update the data? Will it rewrite all data every time?

Or is there a better way to track and apply table schema changes preferably with software development best practices something like table definition as code

Re: Best way to automatically update a delta table schema

Pat — Tue, 01 Nov 2022 09:08:57 GMT

Hi @Harikrishnan P H ,

I wouldn't recommend using `terraform` databricks_table for Unity Catalog Delta Tables. The schema update doesn't work at the moment.

Below error while trying to add/remove column (Delta table):

Error: cannot update table: A table's Delta metadata can only be changed from a cluster or warehouse...

Please check this also:

https://github.com/databricks/terraform-provider-databricks/issues/1624

"so there are quite a few gaps/edge cases with the tables API, hence customers should not use the API or Terraform to create/manage Unity Catalog tables & views at the moment."

Even though, you can create delta tables using terraform, I tested later on from the UI both commands:

show create table;
 
describe table;

they fail to show the output.

thanks,

Pat.

Re: Best way to automatically update a delta table schema

hari — Tue, 01 Nov 2022 09:38:05 GMT

Thanks for the reply @Pat Sienkiewicz .