How does merge schema work

[Constantine]

Let's say I create a table like

CREATE TABLE IF NOT EXISTS new_db.data_table (
  key	STRING,
  value	STRING,
  last_updated_time	TIMESTAMP
) USING DELTA LOCATION 's3://......';

Now when I insert into this table I insert data which has say 20 columns and do merge schema while insertion.

.option("mergeSchema", "true")

So when I display the data it shows me all 20 columns, but now when I look at the table schema through the data tab it still shows only the initial 3 rows i.e. the catalog is not updated.

Wanted to understand how does this work?

[Hubert-Dudek · Accepted solution]

Are you saving to delta file location?

If you append to location, it will not update the hive metastore table as it only updates underlying files. You can use saveAsTable, or you need to refresh the metastore table using the SQL command:

REFRESH TABLE tableName;

[timdriscoll22]

I tried running "REFRESH TABLE tablename;" but I still do not see the added columns in the data explorer columns, while I do see the added columns in the sample data