Overwriting mode do not overwrite

Enrique1987
New Contributor III

I have the following code

Previously I have a delta table with 180 columns in my_path´, I select a column and try to overwrite

  
columns_to_select = ["one_column"] df_one_column = df.select(*columns_to_select) df_one_column.write.format("delta").mode("overwrite").option("mergeSchema", "true").save(my_path) new_schema = spark.read.format("delta").load(my_path).schema target_column = [field.name for field in new_schema.fields] print(len(target_column)) # return 180 

 that returns 180 instead 1, I don understand why and chatgpt 4o neither  thas why I m here.

Thanks in advance, Enrique 

Enrique1987
New Contributor III

ok I get the Issue

  

.option("mergeSchema", "true"

 Is usefull to add more columns, but if you want to reduce columns in your target delta.

Then you need

.option("overwriteSchema", "true"

View solution in original post