cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How does merge schema work

Constantine
Contributor III

Let's say I create a table like

CREATE TABLE IF NOT EXISTS new_db.data_table (
  key	STRING,
  value	STRING,
  last_updated_time	TIMESTAMP
) USING DELTA LOCATION 's3://......';

Now when I insert into this table I insert data which has say 20 columns and do merge schema while insertion.

.option("mergeSchema", "true")

So when I display the data it shows me all 20 columns, but now when I look at the table schema through the data tab it still shows only the initial 3 rows i.e. the catalog is not updated.

Wanted to understand how does this work?

1 ACCEPTED SOLUTION

Accepted Solutions

Hubert-Dudek
Esteemed Contributor III

Are you saving to delta file location?

If you append to location, it will not update the hive metastore table as it only updates underlying files. You can use saveAsTable, or you need to refresh the metastore table using the SQL command:

REFRESH TABLE tableName;

View solution in original post

2 REPLIES 2

Hubert-Dudek
Esteemed Contributor III

Are you saving to delta file location?

If you append to location, it will not update the hive metastore table as it only updates underlying files. You can use saveAsTable, or you need to refresh the metastore table using the SQL command:

REFRESH TABLE tableName;

timdriscoll22
New Contributor II

I tried running "REFRESH TABLE tablename;" but I still do not see the added columns in the data explorer columns, while I do see the added columns in the sample data 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group