Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ07-08-2024 05:07 AM
I have the following code
Previously I have a delta table with 180 columns in my_pathยด, I select a column and try to overwrite
columns_to_select = ["one_column"] df_one_column = df.select(*columns_to_select) df_one_column.write.format("delta").mode("overwrite").option("mergeSchema", "true").save(my_path) new_schema = spark.read.format("delta").load(my_path).schema target_column = [field.name for field in new_schema.fields] print(len(target_column)) # return 180
that returns 180 instead 1, I don understand why and chatgpt 4o neither thas why I m here.
Thanks in advance, Enrique
1 ACCEPTED SOLUTION
Accepted Solutions
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ07-08-2024 06:05 AM
ok I get the Issue
.option("mergeSchema", "true")
Is usefull to add more columns, but if you want to reduce columns in your target delta.
Then you need
.option("overwriteSchema", "true")
1 REPLY 1
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ07-08-2024 06:05 AM
ok I get the Issue
.option("mergeSchema", "true")
Is usefull to add more columns, but if you want to reduce columns in your target delta.
Then you need
.option("overwriteSchema", "true")

