Operations applied when running fs.write_table to overwrite existing feature table in hive metastore
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-07-2023 05:38 AM
Hi,
there was a need to query an older snapshot of a table. Therefore ran:
deltaTable = DeltaTable.forPath(spark, 'dbfs:/<path>')
display(deltaTable.history())
and noticed that every fs.write_table run triggers two operations:
Write and CREATE OR REPLACE TABLE AS SELECT. In both cases operation mode is "append".
Would be interesting to know why two operations are triggered and what does WRITE operation do?
Labels:
- Labels:
-
Feature Table
-
Hive metastore
-
New Table
-
Table