Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-03-2022 03:30 AM
mergeSchema doesn't support all operations. In some cases .option("overwriteSchema", "true") is needed. MergeSchema doesn't support:
- Dropping a column
- Changing an existing column's data type (in place)
- Renaming column names that differ only by case (e.g., “Foo” and “foo”)
more on that topic here https://www.databricks.com/blog/2019/09/24/diving-into-delta-lake-schema-enforcement-evolution.html
My blog: https://databrickster.medium.com/