Overwriting mode do not overwrite

Enrique1987 — Mon, 08 Jul 2024 12:07:45 GMT

I have the following code

Previously I have a delta table with 180 columns in my_path´, I select a column and try to overwrite

  
    columns_to_select = ["one_column"]
    df_one_column = df.select(*columns_to_select)
    df_one_column.write.format("delta").mode("overwrite").option("mergeSchema", "true").save(my_path)
    
    new_schema = spark.read.format("delta").load(my_path).schema
    target_column = [field.name for field in new_schema.fields]
    print(len(target_column)) # return 180

that returns 180 instead 1, I don understand why and chatgpt 4o neither thas why I m here.

Thanks in advance, Enrique

Re: Overwriting mode do not overwrite

Enrique1987 — Mon, 08 Jul 2024 13:05:15 GMT

ok I get the Issue

.option("mergeSchema", "true")

Is usefull to add more columns, but if you want to reduce columns in your target delta.

Then you need

.option("overwriteSchema", "true")

topic Re: Overwriting mode do not overwrite in Data Engineering

Overwriting mode do not overwrite

Re: Overwriting mode do not overwrite