Change schema when writing to the Delta format

[Biber]

Is it possible to reapply schema in delta files? For example, we have a history with field string but from some point, we need to replace string with struct.

In my case merge option and overwrite schema don't work.

[Hubert-Dudek · Accepted solution]

You can change a column’s type or name or drop a column by rewriting the table. To do this, use the

overwriteSchema option. But please back up your table first.

spark.read.table("table_name") \
.write \
.format("delta") \
.mode("overwrite") \
.option("overwriteSchema", "true") \
.saveAsTable("table_name")

[jose_gonzalez]

Hi @Mike Biber​,

Just a friendly follow-up. Did Hubert's response helped you? Let us know if you still need help.

[elgeo]

Hi @Hubert Dudek​. Do you know if this works also for identity columns? Is there another way to do this? The below returns ParseException. Thank you

tt = spark.read.table("table_name") \

.withColumn("ID",col("ID").cast("BIGINT GENERATED ALWAYS")) \

.write \

.format("delta") \

.mode("overwrite") \

.option("overwriteSchema", "true") \

.saveAsTable("table_name")

[Biber]

Hi guys!

Definitely, thank you for your support.

[Hubert-Dudek · Accepted solution]

Please select my answer as the best one.

After the last updates, it is also possible to rename columns using ALTER, but only when mapping is enabled. Check here https://docs.databricks.com/delta/delta-column-mapping.html