- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-30-2022 09:19 AM
Let's say I create a table like
CREATE TABLE IF NOT EXISTS new_db.data_table (
key STRING,
value STRING,
last_updated_time TIMESTAMP
) USING DELTA LOCATION 's3://......';
Now when I insert into this table I insert data which has say 20 columns and do merge schema while insertion.
.option("mergeSchema", "true")
So when I display the data it shows me all 20 columns, but now when I look at the table schema through the data tab it still shows only the initial 3 rows i.e. the catalog is not updated.
Wanted to understand how does this work?
- Labels:
-
Delta table
-
Delta Tables
-
Schema
-
Table schema
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-30-2022 09:30 AM
Are you saving to delta file location?
If you append to location, it will not update the hive metastore table as it only updates underlying files. You can use saveAsTable, or you need to refresh the metastore table using the SQL command:
REFRESH TABLE tableName;
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-30-2022 09:30 AM
Are you saving to delta file location?
If you append to location, it will not update the hive metastore table as it only updates underlying files. You can use saveAsTable, or you need to refresh the metastore table using the SQL command:
REFRESH TABLE tableName;
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-11-2023 12:51 PM
I tried running "REFRESH TABLE tablename;" but I still do not see the added columns in the data explorer columns, while I do see the added columns in the sample data

