- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-22-2022 04:54 AM
Is it possible to reapply schema in delta files? For example, we have a history with field string but from some point, we need to replace string with struct.
In my case merge option and overwrite schema don't work.
- Labels:
-
Delta
-
Delta Files
-
Delta Format
-
Delta table
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-22-2022 05:18 AM
You can change a column’s type or name or drop a column by rewriting the table. To do this, use the
overwriteSchema option. But please back up your table first.
spark.read.table("table_name") \
.write \
.format("delta") \
.mode("overwrite") \
.option("overwriteSchema", "true") \
.saveAsTable("table_name")
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-16-2022 09:15 AM
Please select my answer as the best one.
After the last updates, it is also possible to rename columns using ALTER, but only when mapping is enabled. Check here https://docs.databricks.com/delta/delta-column-mapping.html
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-22-2022 05:18 AM
You can change a column’s type or name or drop a column by rewriting the table. To do this, use the
overwriteSchema option. But please back up your table first.
spark.read.table("table_name") \
.write \
.format("delta") \
.mode("overwrite") \
.option("overwriteSchema", "true") \
.saveAsTable("table_name")
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-03-2022 11:22 AM
Hi @Mike Biber,
Just a friendly follow-up. Did Hubert's response helped you? Let us know if you still need help.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-11-2022 02:32 AM
Hi @Hubert Dudek. Do you know if this works also for identity columns? Is there another way to do this? The below returns ParseException. Thank you
tt = spark.read.table("table_name") \
.withColumn("ID",col("ID").cast("BIGINT GENERATED ALWAYS")) \
.write \
.format("delta") \
.mode("overwrite") \
.option("overwriteSchema", "true") \
.saveAsTable("table_name")
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-09-2022 08:06 AM
Hi guys!
Definitely, thank you for your support.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-16-2022 09:15 AM
Please select my answer as the best one.
After the last updates, it is also possible to rename columns using ALTER, but only when mapping is enabled. Check here https://docs.databricks.com/delta/delta-column-mapping.html
![](/skins/images/582998B45490C7019731A5B3A872C751/responsive_peak/images/icon_anonymous_message.png)
![](/skins/images/582998B45490C7019731A5B3A872C751/responsive_peak/images/icon_anonymous_message.png)