cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Best way to automatically update a delta table schema

hari
Contributor

We have multiple environments where the same tables are added so it's really hard to manually update the schema of the table across all the environments. We know that it's not ideal to update table schema a lot but our product is still evolving and schema changes will occur more frequently than in an ideal case.

I see that databricks has released a databricks_table resource into the public preview. Really excited to see how it evolves.

Does anyone know if terraform stack apply can handle changes to the schema?

If so, How does it update the data? Will it rewrite all data every time?

Or is there a better way to track and apply table schema changes preferably with software development best practices something like table definition as code

1 ACCEPTED SOLUTION

Accepted Solutions

Pat
Honored Contributor III

Hi @Harikrishnan P H​ ,

I wouldn't recommend using `terraform` databricks_table for Unity Catalog Delta Tables. The schema update doesn't work at the moment.

Below error while trying to add/remove column (Delta table):

Error: cannot update table: A table's Delta metadata can only be changed from a cluster or warehouse...

Please check this also:

https://github.com/databricks/terraform-provider-databricks/issues/1624

"so there are quite a few gaps/edge cases with the tables API, hence customers should not use the API or Terraform to create/manage Unity Catalog tables & views at the moment."

Even though, you can create delta tables using terraform, I tested later on from the UI both commands:

show create table;
 
describe table;

they fail to show the output.

thanks,

Pat.

View solution in original post

2 REPLIES 2

Pat
Honored Contributor III

Hi @Harikrishnan P H​ ,

I wouldn't recommend using `terraform` databricks_table for Unity Catalog Delta Tables. The schema update doesn't work at the moment.

Below error while trying to add/remove column (Delta table):

Error: cannot update table: A table's Delta metadata can only be changed from a cluster or warehouse...

Please check this also:

https://github.com/databricks/terraform-provider-databricks/issues/1624

"so there are quite a few gaps/edge cases with the tables API, hence customers should not use the API or Terraform to create/manage Unity Catalog tables & views at the moment."

Even though, you can create delta tables using terraform, I tested later on from the UI both commands:

show create table;
 
describe table;

they fail to show the output.

thanks,

Pat.

hari
Contributor

Thanks for the reply @Pat Sienkiewicz​ .

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.