topic Re: How does merge schema work in Data Engineering

How does merge schema work

Constantine — Wed, 30 Mar 2022 16:19:56 GMT

Let's say I create a table like

CREATE TABLE IF NOT EXISTS new_db.data_table (
  key	STRING,
  value	STRING,
  last_updated_time	TIMESTAMP
) USING DELTA LOCATION 's3://......';

Now when I insert into this table I insert data which has say 20 columns and do merge schema while insertion.

.option("mergeSchema", "true")

So when I display the data it shows me all 20 columns, but now when I look at the table schema through the data tab it still shows only the initial 3 rows i.e. the catalog is not updated.

Wanted to understand how does this work?

Re: How does merge schema work

Hubert-Dudek — Wed, 30 Mar 2022 16:30:07 GMT

Are you saving to delta file location?

If you append to location, it will not update the hive metastore table as it only updates underlying files. You can use saveAsTable, or you need to refresh the metastore table using the SQL command:

REFRESH TABLE tableName;

Re: How does merge schema work

timdriscoll22 — Tue, 11 Jul 2023 19:51:24 GMT

I tried running "REFRESH TABLE tablename;" but I still do not see the added columns in the data explorer columns, while I do see the added columns in the sample data