- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-23-2022 06:00 AM
Guys, good morning!
I am writing the results of a json in a delta table, only the json structure is not always the same, if the field does not list in the json it generates type incompatibility when I append
(dfbrzagend.write
.format("delta")
.mode("append")
.option("inferSchema", "true")
.option("path",brzpath)
.option("schema",defaultschema)
.saveAsTable(brzbdtable))
Failed to merge fields 'age_responsavelnotafiscalpallet' and 'age_responsavelnotafiscalpallet'. Failed to merge incompatible data types LongType and StringType
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-06-2022 05:39 AM
Hi @Tássio Santos
The delta table performs schema validation of every column, and the source dataframe column data types must match the column data types in the target table. If they don’t match, an exception is raised.
For reference-
https://docs.databricks.com/delta/delta-batch.html#schema-validation-1
you can cast the column explicitly before writing it to target table to avoid this
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-06-2022 05:39 AM
Hi @Tássio Santos
The delta table performs schema validation of every column, and the source dataframe column data types must match the column data types in the target table. If they don’t match, an exception is raised.
For reference-
https://docs.databricks.com/delta/delta-batch.html#schema-validation-1
you can cast the column explicitly before writing it to target table to avoid this
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-25-2022 02:06 AM
The following example shows changing a column type:
(spark.read.table(...)
.withColumn("birthDate", col("birthDate").cast("date"))
.write
.mode("overwrite")
.option("overwriteSchema", "true")
.saveAsTable(...)
)
Details see https://docs.databricks.com/delta/update-schema.html

